Testfx: Tests fail to run on build server with V1.1.17, but will run with V1.1.14

[robinvanpoppel]

Description

I recently upgrade a solution from MSTest V1 to MSTest V2 with version 1.1.14 of the TestAdapter and the TestFramework. Based on that experience i decided to upgrade two other projects, but this time with 1.14 of the TestAdapter and the TestFramework.

The new projects runned locally on VS2017, but running the unit tests on a VS2013 build server fails with the following exception:
An exception occurred while invoking executor 'executor://mstestadapter/v2': Constructor on type 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner' not found.

Downgrading the packages from 1.1.17 to 1.1.14 fixed the issue for me.

Steps to reproduce

Install MSTest V2 version 1.1.17 for both the TestAdapter and TestFramework.
Queue a build to a VS2013 build server.

Expected behavior

The tests run locally on VS2017 and on the build server

Actual behavior

The unit tests are not run with the following message in the build log.

An exception occurred while invoking executor 'executor://mstestadapter/v2': Constructor on type 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner' not found.

Environment

Build server: VS2013
Tests fail with version 1.1.17 . Rolling back to 1.1.14 fixes the issue.

[AbhitejJohn]

@robinvanpoppel : It looks like you have projects targeting multiple versions of the Framework and adapter - 1.1.14 and 1.1.17. Can you move all test assets to use one version, clean up all project output assets and retry this please?
If the CI system has a cache of test adapter binaries please make sure to clean that up/upgrade them as well.

[robinvanpoppel]

@AbhitejJohn I'm sorry if it wasn't clear from the description above.

At all times i referenced the same versions for the framework and the adapter. The build succeeds when I use 1.1.14 for both of these packages. It fails when I upgrade both of them to 1.1.17

[AbhitejJohn]

ah ok. Then this should be the case of stale adapter assemblies in the build. I'm assuming this is a XAML Build definition. Were the adapter assemblies checked in(to provide the custom test adapters to the runner)?

[WallyDoodlez]

I have the same issue. I upgraded from 1.1.14 to 1.1.17 with Nuget and it works fine locally with VS2015U3 but when it is ran by an TFS Test agent the test always fails with

An exception occurred while invoking executor 'executor://mstestadapter/v2': Constructor on type 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner' not found.

I have double checked and made adjustments to all references to testframework.dll testframework.extensions.dll and testadapter.targets testadapter.props in all csproj files.

I looked at the compile output of the TFS Test agent, all references are ok and has the same digital signature as the one I have locally, and yes this is from a XAML build definition

[AbhitejJohn]

@WallyDoodlez : How are you running these tests in the XAML build? via a test adapter path specification detailed here? If yes, these adapter assemblies that were checked in need to be upgraded as well.

[WallyDoodlez]

No, its just a normal unit test that I create with the MSTestV2 framework.

[AbhitejJohn]

ok. Can you try after restarting the test agent service please? It seems to be caching the older version of the adapter and re-using it for all the runs. We will get a local repro out to investigate this.

[RobSiklos]

@AbhitejJohn We restarted the entire build server, and the problem persists.

[AbhitejJohn]

Thanks @RobSiklos . We are investigating this.
Tagging @jayaranigarg to help.

[binaryWard]

I have ran into this same bug. The following is how I have encountered it.

• Solution contains 1 project Unit Test project. It is set to target .NET 4.6.2.
• The unit test project and unit tests worked as expected with 1.1.11
• Using NuGet upgraded from 1.1.11 to 1.1.17 and see the same error.
• Have confirmed a solution with 1 unit test project with 1.1.14 work. May need to restart VS and delete bin/
• A solution that has 3 projects console, assembly, unit test project at version 1.1.14 does not work.
• Test Explorer will not list unit tests.
• Unit tests will not execute context menu on a specific unit test method.
• If you have the Test Explorer open before upgrading to the newer version the tests will be listed but will not execute. Close/Open VS2017 the Test Explorer will not display the tests.

The Tests output error when I have 3 projects console, assembly, and unit test. All 3 target 4.6.2 and the version 1.1.14 installed is "The given assembly name or codebase was invalid. (Exception from HRESULT: 0x80131047)"

Changing the unit test project to 1.1.11 when from 1.1.14 or 1.11.17 does not fix the problem. I have to remove and create a new unit test project. I have checked the project references and the nuget package metadata and files. All appears correct when the version is changed. I still have to recreate the project.
Was able to reproduce it twice already.

The console and assembly projects have BenchmarkDotNet.0.10.5 package added to them. When I added it the options for use the latest dependency version was selected.
The Unit Test project references the assembly project.

Windows 10 pro - Creative update installed
Microsoft Visual Studio Community 2017
VisualStudio.15.Release/15.1.0+26403.7
Version 4.7.02046

Visual Basic 2017 00369-60000-00001-AA097
Visual C# 2017 00369-60000-00001-AA097
Visual C++ 2017 00369-60000-00001-AA097
Visual F# 4.1 00369-60000-00001-AA097
ASP.NET and Web Tools 2017 15.0.30320.0
ASP.NET Web Frameworks and Tools 2017 5.2.50303.0
Azure App Service Tools v3.0.0 15.0.30209.0
Azure Data Lake Node 1.0
Azure Data Lake Tools for Visual Studio 2.2.5000.0
Common Azure Tools 1.9
JavaScript Language Service 2.0
JavaScript Project System 2.0
JavaScript UWP Project System 2.0
KofePackagePackage Extension 1.0
Microsoft Azure Hive Query Language Service 2.2.5000.0
Microsoft Azure Tools - v2.9.50131.1
Microsoft MI-Based Debugger 1.0
Microsoft Visual Studio VC Package 1.0
Node.js Tools 1.3.50316.00
NuGet Package Manager 4.1.0
Redgate SQL Search 2.4.1.1284
SQL Server Data Tools 15.1.61702.140
ToolWindowHostedEditor 1.0
TypeScript 2.1.5.0
Visual Studio tools for CMake 1.0
Visual Studio Tools for Unity 3.1.0.0
Visual Studio Tools for Universal Windows Apps 15.0.26403.07
WiX Toolset Visual Studio Extension 0.9.17.13693

[jayaranigarg]

@RobSiklos : When you are upgrading to v1.1.17, can you make sure that the older version(v1.1.14) is indeed removed as part of your check-in? If you check-in using VS IDE, then you will observe that the changes of deleting older Nuget Packages are by default under "Excluded changes" and you will need to promote these changes so that they reflect under "Included changes" (under Pending changes tab) and then do a check-in.

Also, please make sure that all the TestProjects in your code points to latest version of Nuget Packages(v1.1.17). Some TestProjects pointing to older version and some pointing to newer version can also cause this issue.

[Ytrog]

Same issue: when referencing 1.1.17 it runs locally, but not on the server and when referencing 1.1.14 it runs on the server, but not locally. On the server it still gives "An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified." when downgraded from 1.1.17 to 1.1.14.

EDIT
Using 1.1.14 together with the 1.* version of the test task and not the 2.* (preview) task worked.

[RobSiklos]

@jayaranigarg I did a fresh checkout of the source tree, and used grep to search for any files containing "1.1.14", and nothing came up.

[jayaranigarg]

On server side -
The issue is occurring because somewhere there is a stale copy of v1.1.14 package on the server which is messing up with v1.1.17
Workaround is to just find that stale Nuget package folder(v1.1.14) and remove it. Mostly you will find this Nuget folder at solution level under "/packages".
Also, in future whenever you do an upgrade for Nuget packages, make sure that the older version of the packages are deleted cleanly.

On Client side(i.e. locally) -
Close VS IDE. Try clearing "%temp%\VisualStudioTestExplorerExtensions" and reopen VS. Clean and rebuild solution. Things should just work.

[jayaranigarg]

@RobSiklos : Did you try searching for folders having names like "1.1.14" ?

[RobSiklos]

@jayaranigarg We did not actually delete the Nuget package folder for 1.1.14, but I don't see how this should matter, since it's not referenced by any of the projects in the solution.

[jayaranigarg]

@RobSiklos : In XAML workflow, we search for adapter dlls in the "/packages" folder and loads the first matching dll we find. We do not further load any other matching adapter dll as one has already been loaded and we cannot load another one because they have same assembly versions.
Not deleting the package folders for 1.1.14 may mean that dlls gets loaded from 1.1.14 Nuget package (in parent appdomain) and we will have objects of types defined in 1.1.14. Then, when we create a separate child appdomain for execution and pass on these v1.1.14 objects, the child appdomain loads adapter dlls from bin/debug which contains v1.1.17 dlls and hence bombs (expecting objects of types defined in 1.1.17 but gets objects of types defined in 1.1.14).

This issue is specific to XAML workflow, as one needs to explicitly check-in "/packages" folder.

We would suggest to delete the 1.1.14 version of Nuget packages for a workaround as of now. We will try investigating better options for the same that do not involve explicit deletion.

[cytron201]

"An exception occurred while invoking executor 'executor://mstestadapter/v2': Constructor on type 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner' not found."

I am also getting this issue locally after moving to 1.1.17. All projects are 4.6.2 and created in VS2017. have completely deleted all code and pulled down fresh from TFS and still see the issue, so it's not to do with leftover 1.1.14 refs.

Have checked xml of all project files and confirm they are pointing to 1.1.17.

[AbhitejJohn]

@cytron201 : Since this is something you are hitting locally when running from VS, could you clean out all the cache folders listed here, restart VS and give it a shot please?
you could also set VS_UTE_Diagnostics system level environment variable to 1 to get diagnostic logs indicating where adapters are being loaded from. These would show up in the Output -> Tests pane of VS.

[christophedemey]

Same thing here, just downgrading to .14 resolves my issue, and this is on my visual studio 2017, not a build server.

[AbhitejJohn]

@christophedemey : I guess you already tried out cleaning the cache as the above comment suggests. Can you turn on the environment variable above and put out the logs from Output->Tests pane here please?

[christophedemey]

@AbhitejJohn no need, its verry easy to re-create the issue, i just did it on my PC @ home.
Using vs2017

• Create new console app (.NET 4.6.2).
• Add unit test project.
• Run the default unit test (which contains no code so succeeds)
• Update packages (from v11 to v17).
• Run test again.

Error :

------ Discover test started ------
========== Discover test finished: 1 found (0:00:01,3989957) ==========
------ Run test started ------
An exception occurred while invoking executor 'executor://mstestadapter/v2': Constructor voor type Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner is niet gevonden.
========== Run test finished: 0 run (0:00:00,4959993) ==========

is niet gevonden ofcourse means not found :)

[AbhitejJohn]

@christophedemey : Can you try after restating your VS instance please? There is a caching issue in Test Explorer which uses the old adapter even after upgrade. A new instance should re-create this cache.

[christophedemey]

@AbhitejJohn this resolves it, how silly :-D
Maybe a ps script is required with the nuget package to resolve this ?
Or maybe a readme which states "REBOOT YOUR VS" :)

[MichaelGiger]

I get the same Error with 1.1.17. Restarting VS instances solved the problem locally, but not on build server! For the moment, it works with:
MSTest.TestAdapter.1.1.12
MSTest.TestFramework.1.1.11

[AbhitejJohn]

@christophedemey : This is something that would need to be fixed on the Test Explorer side. We are tracking to fix this in a future update of VS.
@MichaelGiger : Were you able to try out the steps that @jayaranigarg shared above?

[AbhitejJohn]

@RobSiklos, @robinvanpoppel @Ytrog : I'm hoping this is no longer an issue and can be closed. Do let us know otherwise.
We will follow up on fixing the caching issue in VS Test Explorer as part of #117.

[RobSiklos]

@AbhitejJohn : I wouldn't consider the issue to be closed, because I don't think that the XAML workflow should care about the packages folder. It should only use the adapter/framework that's referenced by the test project assemblies.

[AbhitejJohn]

I agree. However AFAIK there aren't any plans to make changes in this workflow anytime soon. This should ideally restore referenced packages on build and not require folks to check them in. I'm not really sure if we could just add in a pre-setup step that does a restore before build instead. Tagging @nigurr and @acesiddhu to see if there is anything better that can be done here.

[waratah]

I had this problem after upgrading a few days ago. I just got offered a new release of VS community, downloaded and installed. Test now works correctly. NB: I did upgrade by test packages to mstest.** packages to 1.1.17 but this did not resolve the issue at the time.

[datvm]

Confirm too, updating Test package through Nuget broke it. Restarting VS did fix it.

[Mertsch]

I am not having the explicit error message of
An exception occurred while invoking executor 'executor://mstestadapter/v2': Constructor voor type Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.UnitTestRunner is niet gevonden.
in every case, but also for me 1.17 often fails to discover tests completely
Strangely it only does so inside of VS 2017 15.2, the build server (or starting them via vstest directly) run the test just fine.

On the first ever discovery (delete all packages and build output) I get some errors with 1.17 which say something like (text is me, remembering it)

ArgumentNullException parameter name: solution
solution can't be null
ArgumentNullException parameter name: solution
solution can't be null

(yes, twice) in the "Tests" output window

[AbhitejJohn]

@Mertsch : The errors on solution being null would be coming from the Test Explorer code. This shouldn't be related to the adapter in use. Can you share the diagnostic logs in another issue and I'l route the right folks in. Here is how you turn them on.

[Mertsch]

@AbhitejJohn Thanks for the hint, doing the cache clear (see https://github.com/Microsoft/testfx/issues/177#issuecomment-299615047) seems to have fixed the issue.
Specifically %temp%\VisualStudioTestExplorerExtensions, cause I already did all the other 3 places

[Ytrog]

Am not in the position to test this for some time (no access to that server anymore bc new job), so if anyone else that did have the problem can confirm the fix it's fine by me.

[acesiddhu]

@RobSiklos does this link help ?
https://docs.microsoft.com/en-us/nuget/consume-packages/team-foundation-build

[AbhitejJohn]

Thanks @acesiddhu . Closing this thread in favor of the link above on restoring nuget packages instead of checking them in, which seems to be causing version mismatches.

[ryanwebjackson]

Not sure if this is welcome, since the issue is Closed, but I just encountered this issue with Visual Studio 2017 Enterprise Update 2. I went with version 1.1.14 since it seems to work without issue.

[AbhitejJohn]

@ryanwebjackson : I'm guessing this is not on a TFS instance. For VS 2017 client, here is the workaround: https://github.com/Microsoft/testfx/issues/177 . Essentially a restart of the VS instance should fix this.

[eakoning]

In my case consolidating multiple versions of the MSTest.TestAdapter and MSTest.TestFramework NuGet Packages to the latest version solved it.

[softlion]

+1 on 27/6/2017

[Izzmo]

+1 .. not working with v1.1.18, but still works with v1.1.14.

[steinmr]

Our cause for this error is slightly different. We use VSTS build (not XAML) to build a bunch of solutions in a single task. The problem was that one of the solutions used 1.1.1 and another used 1.1.120. When the tests started for the second solution, it searched for the testadapter in the root folder shared by all solutions, and it found the testadapter of the first solution, and used that. Our fix was simple, upgrade the first solution to 1.1.20

[CiprianSavoiu]

binaryWard solution worked for me. Updated the two unit test project with version 1.20 from 1.11 and then deleted the content of debug folders. After VS2017 restart everything worked fine.
This solved also the OUTPUT option not available on 1.11

[justsurvey]

you can add dll "Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll"