Testfx: An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'

[northtyphoon]

Description

The test output includes "System.IO 4.1.2.0" and the following binding redirect in TestAssembly.dll.config

<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  <dependentAssembly>
    <assemblyIdentity name="System.IO" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.2.0" newVersion="4.1.2.0" />
  </dependentAssembly>
</assemblyBinding>

The test passed in Visual Studio. However, the test failed in the build machine as the following error.

An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies

I also notice if I manually removed TestAssembly.dll.config from the output folder, the test started to pass in the build machine.

Steps to reproduce

In VSTS build, run the latest Visual Studio Test task.

Expected behavior

Test pass.

Actual behavior

2017-10-19T23:47:12.3630779Z Starting test execution, please wait...
2017-10-19T23:47:19.4800462Z Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2017-10-19T23:47:19.4800462Z

Environment

Operating system : WS2016
VS2017 15.4.1
Visual Studio test runner 2.1.15

Package version of MSTest:
"MSTest.TestAdapter" Version="1.2.0"
"MSTest.TestFramework" Version="1.2.0"

[kant2002]

If this is just some binding redirect issues, it will greatly benefit from increased logging with information which test assembly could not be loaded, since that alone greatly speedup troubleshooting. I hit similar issue in the past where binding redirects was required by the test.

[Izzmo]

@kant2002

it will greatly benefit from increased logging with information which test assembly could not be loaded

Okay... you might want to enlighten us how one would do this :)

[AbhitejJohn]

To get more logs from the CI pipeline, could you set System.Debug in the Build definition to true and share the build logs with us please?

If you could repro this locally using vstest.console, then it would also help to enable vstest.executionengine*.exe.config logging. Here is how you can enable that.

I think we need to add a troubleshooting section in testfx-docs to help get more logging.

[codito]

I think we need to add a troubleshooting section in testfx-docs to help get more logging.

Added docs for collecting traces in CI task here: https://github.com/Microsoft/vstest-docs/blob/master/docs/diagnose.md#collect-trace-in-vsts-ci-vstest-task

[northtyphoon]

Thanks for the troubleshooting guide to collect detail trace. I will try to get more logs and post back later.

[northtyphoon]

I enabled system.debug.

2017-10-31T21:44:42.7837155Z Microsoft (R) Test Execution Command Line Tool Version 15.0.26929.2
2017-10-31T21:44:42.7837155Z Copyright (c) Microsoft Corporation. All rights reserved.
2017-10-31T21:44:42.7837155Z
2017-10-31T21:44:43.2496697Z Starting test execution, please wait...
2017-10-31T21:44:47.4338786Z Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.
2017-10-31T21:44:47.4338786Z
2017-10-31T21:44:47.8549250Z
2017-10-31T21:44:47.9249367Z Information: Additionally, you can try specifying '/UseVsixExtensions' command if the test discoverer & executor is installed on the machine as vsix extensions and your installation supports vsix extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
2017-10-31T21:44:47.9249367Z
2017-10-31T21:44:47.9489567Z ##[debug]rc:1
2017-10-31T21:44:47.9489567Z ##[debug]success:false
2017-10-31T21:44:47.9509367Z ##[debug]rm -rf E:\A_work_temp\b3938700-be84-11e7-91e2-b52a751f8768.runsettings
2017-10-31T21:44:47.9509367Z ##[debug]removing file
2017-10-31T21:44:47.9509367Z ##[warning]Vstest failed with error. Check logs for failures. There might be failed tests.
2017-10-31T21:44:47.9509367Z ##[debug]Processed: ##vso[task.issue type=warning;]Vstest failed with error. Check logs for failures. There might be failed tests.
2017-10-31T21:44:47.9509367Z ##[debug]Release.ReleaseUri=undefined
2017-10-31T21:44:47.9509367Z ##[debug]Release.ReleaseId=undefined
2017-10-31T21:44:47.9509367Z ##[debug]Build.BuildUri=vstfs:///Build/Build/1101004
2017-10-31T21:44:47.9509367Z ##[debug]Build.Buildid=1101004
2017-10-31T21:44:47.9509367Z ##[debug]Agent.Version=2.120.2
2017-10-31T21:44:47.9509367Z ##[debug]telemetry area: TestExecution feature: TestExecutionTask data: {"builduri":"vstfs:///Build/Build/1101004","buildid":"1101004","areacode":"ExecuteVsTest","result":"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1","tracepoint":1005,"isusererror":true}
2017-10-31T21:44:48.4491707Z ##[debug]Processed: ##vso[telemetry.publish area=TestExecution;feature=TestExecutionTask;]{"builduri":"vstfs:///Build/Build/1101004","buildid":"1101004","areacode":"ExecuteVsTest","result":"C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1","tracepoint":1005,"isusererror":true}
2017-10-31T21:44:48.4491707Z ##[error]Error: C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe failed with return code: 1

[jayaranigarg]

@northtyphoon : Can you check which version of System.IO.dll is present in TestRun folder in both machine(one with VS installed and build machine) and let us know.

Also, instead of hard-coding the assembly binding redirects in app.config, can you add this in your *.csproj

<PropertyGroup>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>
</PropertyGroup>

This will generate a *.dll.config which will have the appropriate assembly binding redirects. See if things work for you after this change.

[Mertsch]

@JayJanarthanan I have two different Test-Projects both with automatic binding redirect.
Both do copy the same System.IO.dll to their output directory.
Both have the following binding redirect

<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  <dependentAssembly>
    <assemblyIdentity name="System.IO" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.1.2.0" newVersion="4.1.2.0" />
  </dependentAssembly>
</assemblyBinding>

One works fine, one has the error mentioned above

2017.11.01 14:40:20.779   ERROR An item with the same key has already been added.
2017.11.01 14:40:20.832   ERROR An exception occurred while invoking executor 'executor://mstestadapter/v2': Could not load file or assembly 'System.IO, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified.

[jayaranigarg]

@Mertsch : Can you please share repro projects with us?

[Mertsch]

@jayaranigarg No I can not, but I found the root cause for this issue.
There is a difference between the normal project output and DeploymentItem output.

I have a fairly complex solution with a lot of dependencies on other projects and frameworks, like the .NET Standard 2.0 library and EF Core (from .NET Framework 4.6.1 projects) and the <project>\bin\Debug folder of set projects contain all required .dlls from all dependencies.

But whenever I specify any DeploymentItem, my <SolutionDir>\TestResults\Deploy_User 2017-11-02 18_24_42\Out contains only a limited set of dependencies.

The System.IO is simply missing from the \Out dir.

[Mertsch]

@jayaranigarg I was now able to nail down the issue
Demo repo is here https://github.com/Mertsch/testfxIssues295

So what is the issue?

In my demo I have referenced a .NET Standard 2.0 library https://www.nuget.org/packages/fm.Extensions.Common/
this causes the project output (\bin\Debug) to contain System.IO.dll and the appropriate binding redirects. Note these references are added implicitly! There is no direct reference to System.IO and they are only added to the project output because a .NET Standard 2.0 library is referenced.

If you now specify any DeploymentItem, the \TestResults\Deploy\Out directory contains an incomplete set of assemblies, but still retains the binding redirects.

Which leads to NormalTests to run fine on their own

But together with DeploymentItem, they all fail

[jayaranigarg]

@Mertsch : Can you move to latest Visual Studio Preview and see if it works for you...
It is working fine for me with latest VS installed..

[Mertsch]

@jayaranigarg No, I will stay with the release version (15.4.2), but report back as soon as 15.5 is out

[AndersMalmgren]

I have ran into this exception too on our build server and local when not using resharper test runner.
System.IO.dll is present in Bin/debug, but its not presented in the testrunner folder. So it fails to copy

.NET Core 2.0 project targeting net47. TestFramework and TestAdapter at 1.2

Any thoughts?

[AndersMalmgren]

Ok, so auto binding redir. fails to add System.IO into the resulting app.config in output folder

  <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
  <GenerateBindingRedirectsOutputType>true</GenerateBindingRedirectsOutputType>

[AndersMalmgren]

Updated to 15.4.4, no dice

[jayaranigarg]

@AndersMalmgren : Can you please explain your scenario in detail and share your repro project with us?
If you have DeploymentItem attribute on tests, in that case we do not copy System.*.dll to testrun folder and they are actually picked up from GAC at runtime.

[AndersMalmgren]

We have the attribute, if I add DeploymentItem(@"System.IO.dll") it works. Is this by design?

[AndersMalmgren]

My workaround above works on my machien, but when build server is building and running tests I get

Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Method not found: 'System.String Microsoft.VisualStudio.TestTools.UnitTesting.IgnoreAttribute.get_IgnoreMessage()'.

[AndersMalmgren]

THere is a ton of missing files in the Out folder, here you see the out folder on the left and bin/debug on the right.

Then it takes from the GAC which is of wrong version adn I get above message. (Is my guess). So how can i get it to includde everything in build folder?

[AndersMalmgren]

I thought I had created a repro, net47 target platform plus DeploymentItem attribute, the repro fails inside our sln, but when I move the repro to a stand alone blank sln it does not fail anymore. So there is a third unknown factor here

[jayaranigarg]

@AndersMalmgren : What do you mean by ".NET Core 2.0 project targeting net47" ? Did you mean that you created a .net core test project and changed the target framework to net47 OR you created a .net core test project and multi-targeted it to netcoreapp2.0 and net47 ?

Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Method not found: 'System.String Microsoft.VisualStudio.TestTools.UnitTesting.IgnoreAttribute.get_IgnoreMessage()' usually happens when adapter and framework versions mismatch. Make sure all the projects in your solution are using same version of both framework and adapter i.e.1.2.0 (in your case).

Why exactly are you trying to run your tests from a different location(i.e. Out folder) and why not from bin/Debug?

I think there is some confusion in the concept of [DeploymentItem] attribute. It is generally used when you want to sandbox your tests to run from a particular location deploying very specific versions of dlls and other dependencies. In your case, I think you can run tests from bin/Debug itself, and drop any required files to bin/Debug itself by setting CopyLocal to true.
PS: [DeploymentItem] is a no-op in case of netcore projects.

[AndersMalmgren]

I'm not trying to run it outside of bin/debug. mstest does that when it detects a DeploymentItem attribute. But if I create a blank solution and add a .net core test project targeting net47 I do not get the same behavior. Then its run under bin/debug and everything works. Its a third factor at play which I cant find.

[jayaranigarg]

@AndersMalmgren : Instead of using DeploymentItem attribute in your solution, can you instead set CopyLocal to true for the files that you need to get dropped in bin/Debug? Effectively we want to check if tests are running fine from bin/Debug for your original solution.

Can you edit your csproj (i.e. .net core test project targeting net47 , the one mentioned in above comment), and let us know what value is present in value tag ?

[AndersMalmgren]

@jayaranigarg The DeploymentItem attribute is no longer needed since it run correctly under bin/debug. So it solves my problem (DeploymentItem was needed with the old mstest runner). But it seems you have a bug. If DeploymentItem is present, it runs under "Out" instead and alot of files are not copied. Not only System.* but others too. Like my screenshot above suggests.

I will check in my removal of DeploymentItem and see if it runs also on our build server agent (visualstudio.com)

[AndersMalmgren]

Nope, it works on my machine, but not on the build agent

Error: An exception occurred while invoking executor 'executor://mstestadapter/v2': Method 'get_Arguments' in type 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.Execution.TestMethodInfo' from assembly 'Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' does not have an implementation.

[jayaranigarg]

@AndersMalmgren : This exception comes when there is a mismatch in framework and adapter version. Can you make sure both nuget packages i.e. MSTest.Adapter and MSTest.Framework have same version?

[AndersMalmgren]

Sorry for late reply. I revisited this problem this morning and found the problem. It was a hard catched problem on our part.

We have a sister solution to the solution holding my unit tests, this sister solution had 1.1.17. Since its a sister solution it shares the package (nuget) folder. We have another unit test project that also builds and runs on the build server. This project referenced 1.1.17 directly not using nuget, probably a dev used resharper and by mistake added the reference not the nuget package (Not me!! :D). Problem was the error happened in my unit test project not the actual project, plus that unit test project did not even turn up when managing nuget packages so it was hard for me to catch. First when I updated the sisterproject and 1.1.17 folder was removed from packages folder did the solution holding my project fail to build and I could correct it to use 1.2.

Ah, this reminds me of when I was a young dev and we had com dlls :P aka dll hell

[jayaranigarg]

Great!!
If your issue is resolved, can we go ahead and close this thread?

[jayaranigarg]

Closing the thread. Please feel free to re-open this, in case you face the issue again.

[tfabraham]

There is still an issue here, related to the presence of [DeploymentItem]. I've been fighting the same issue as the original issue for hours today, using v1.2.0. After simply deleting the [DeploymentItem] lines, the problem disappeared!

If [DeploymentItem] can no longer be used without causing problems, then it either needs to be fixed, or there needs to be clear documentation about avoiding it in this situation. I wasted half my day on this, and another half day a few weeks ago. I would have lost much more time had I not found this issue. Thanks.

[jayaranigarg]

@tfabraham : There is some confusion in the community about usage/functionality of [DeploymentItem] attribute, we will consider adding a documentation for same.

[JP-CZ]

I want to rate this up as this is preventing us from running unit tests on some machines. We have a test project which references a dotnetcore 2.0 assembly, along with other dotnet fw 4.6.1 assemblies. Is there a solution for this problem? Anything to solve this assembly redirect thing? It seems there is another addition to the DLL hell :)

[AndersMalmgren]

As a senior system architect I can tell you this @jayaranigarg, when a SDK steals hours of a dev's time something is fundamental wrong with the design of it. And it seems many are coming here, you need to make sure this errors is fixed, or provide better log info, or make sure in other ways people do not spend hours and hours on it.

[jayaranigarg]

@JP-CZ : Did you try adding lines mentioned in this comment in your *.csproj https://github.com/Microsoft/testfx/issues/295#issuecomment-341080764

Do you also have [DeploymentItem] attribute on your tests? If so, then instead of using DeploymentItem attribute in your solution, can you instead set CopyLocal to true for the files that you need to get dropped in bin/Debug? Effectively we want to check if tests are running fine from bin/Debug for your original solution.

If possible, can you share a repro project with us?

[jayaranigarg]

@AndersMalmgren : Thank you for your feedback. We will surely try to improve upon it.

[JP-CZ]

Yes we added these lines in all of our projects. It clearly caused by the DeploymentItem attribute on the test method. This worked until now. Problem is we have many tests using this attribute to provide test data during test execution. Is this pattern no longer supported? Is unclear to me how this attribute effects what kind of files are deployed or where tests are run.

[jayaranigarg]

@JP-CZ : Until when was this working for you and after what change/upgrade did this stop working?

Normally, tests are ran from bin/Debug folder. If you want some of your project files to be dropped in bin/Debug, then simply setting CopyLocal to true does the job.

[DeploymentItem] attribute is needed when you want to sandbox your tests to run from a particular location(other than bin/Debug) deploying very specific versions of dlls and other dependencies. In this case, not all the files present in bin/Debug are deployed to the new testrun location and are picked up from VS install locations.

[tfabraham]

@jayaranigarg: you keep telling us either 1) not to use DeploymentItem or 2) that it's needed when you want to sandbox your tests... but when DeploymentItem is present anywhere in the test suite, the entire suite is unusable due to the binding issues. This is not just a documentation issue, there is a bug that needs to be fixed.

There are thousands of test suites out there using DeploymentItem that are upgrading to the new project format and starting to pull in .NET Standard dependencies. Due to no fault of our own, they simply blow up with these obscure errors after the upgrade. I hope that is not an acceptable outcome to your team!

[jayaranigarg]

@tfabraham : Apologies for the inconvenience. Let us dig deeper in this and figure out a better solution/fix for this issue.

[adamdriscoll]

I just encountered this issue as well. 1.5.5 VS and 1.3.0-beta2 of the test adapter\test framework

[EwaldStricker]

Same issue solved by using as reference

• "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\ReferenceAssemblies\v4.0\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll" Version 10.0.0.0
  not
• "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\ReferenceAssemblies\v2.0\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll" Version 10.1.0.0

Or solved by uninstalling package

• MSTest.TestAdapter and MSTest.Testframework Version "cant remember"
  and install package
• MSTest.TestAdapter and MSTest.Testframework Version 1.1.18

[ivonin]

So, is there a work-around so we can use the 1.3.0-beta2 version? I have the same issue after trying to upgrade my project to this version of Framework and Adapter packages.

Upgrade did generate a bunch of binding redirects, but they don't seem to help the issue.

[ivonin]

Hmm, removing the bindingredirects seems to work as a work-around.

[EwaldStricker]

Workarround:

• Solution 1 from
  https://msdn.microsoft.com/en-us/library/ms182475.aspx ->using Post Build Scripts

[cltshivash]

The problem of binding redirects getting added is similar to issue #241 . We are following up with the Nuget team on this on the fix to be done. There's another issue when the deploymentitem attribute is used in which we do a selective copy of the dlls. We will fix that issue by copying whatever is present in the output location of the test source.

[cltshivash]

@AndersMalmgren @tfabraham

There are two issues in this case.
1 -> With assembly redirects present in the app.config of a test project the redirected assembly is not present in the output folder of the test assembly leading to a version being picked from GAC which then fails. This was followed up on the nuget repo and the fix for this issue is to use package references. With VS 15.7 (right click on Project -> References should give you the option) Please check if this helps.

For more details please refer to the issue on the nuget repo.
NuGet/Home#6723

2 -> The set of dlls being used is subset of the ones used from bin/debug. We have a PR which will copy the content of bin/debug to the test deployment folder https://github.com/Microsoft/testfx/pull/391

[zootyzoot]

I was able to solve this in my project by forcing Visual Studio to not use testhost.x86.exe

To do this,

1. open the Test menu from the toolbar
2. select Test Settings,
3. click on Default Processor Architecture
4. select x64

If you run vstest.console.exe by command-line you can send /Platform:64

[cltshivash]

@AndersMalmgren @tfabraham Please let us know if the provided workaround helps.

[EwaldStricker]

hi thanks for your support,
i cant get back with a feedback soon for a uncertain time ....
sorry for that
i leave a task for me in my list

kr ewald

Freundliche Grüße

i. A. Ewald Stricker

Software Engineer Informationstechnologie Eigenentwicklung
Daten, Modelle & Handel (C-TIHD)

EnBW Energie Baden-Württemberg AG
Durlacher Allee 93 · 76131 Karlsruhe
Telefon +49 721 / 63-15429
Mobil +49 174 / 3005967
mailto: e.[email protected]
www.enbw.com

20.000 Mitarbeiter. Ein Versprechen: Wir machen das schon.
www.enbw.com/WirMachenDasSchon

EnBW Energie Baden-Württemberg AG
Sitz der Gesellschaft: Karlsruhe · Amtsgericht Mannheim · HRB Nr. 107956
Vorsitzender des Aufsichtsrats: Lutz Feldmann
Vorstand: Dr. Frank Mastiaux (Vorsitzender), Dr. Bernhard Beck,
Thomas Kusterer, Dr. Hans-Josef Zimmer

Am 04.06.18 um 10:47 schrieb Shiva Shankar

@AndersMalmgren @tfabraham Please let us know if the provided workaround helps.

--
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/Microsoft/testfx/issues/295#issuecomment-394280227

[MisinformedDNA]

I'm hitting this issue as well in VSTS. All my projects use the .NET Core project format. PackageReference is used everywhere. The one project that fails is the one with DeploymentItems. I was using the VsTest task and it kept failing. I switched to dotnet test and it passed. Maybe that will help someone.

[rossng]

This caveat should be documented here, the first search result for DeploymentItem.