Benchmarkdotnet: [Bug] FileNotFoundException On 0.9.4.

[ig-sinicyn]

Fresh VS 2015u1 install + latest Resharper EAP (if important at all). No additional extensions/packages.

New Console App project, add BenchmarkDotNet nuget package, copy the code from getting started, run.

Fails with:

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Build.Framework, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. Не удается найти указанный файл.
File name: 'Microsoft.Build.Framework, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'
   at BenchmarkDotNet.Toolchains.Classic.ClassicBuilder.Build(GenerateResult generateResult, ILogger logger, Benchmark benchmark)
   at BenchmarkDotNet.Running.BenchmarkRunner.Build(ILogger logger, IToolchain toolchain, GenerateResult generateResult, Benchmark benchmark)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(Benchmark benchmark, ILogger logger, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(IList`1 benchmarks, ILogger logger, String title, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(IList`1 benchmarks, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run[T](IConfig config)
   at Bench.Test.Program.Main(String[] args) in c:\users\igors\documents\visual studio 2015\Projects\Bench.Test\Program.cs:line 19
   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

=== Pre-bind state information ===
LOG: DisplayName = Microsoft.Build.Framework, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
 (Fully-specified)
LOG: Appbase = file:///c:/users/igors/documents/visual studio 2015/Projects/Bench.Test/bin/Debug/
LOG: Initial PrivatePath = NULL
Calling assembly : Microsoft.Build.Utilities.v12.0, Version=12.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in default load context.
LOG: Using application configuration file: c:\users\igors\documents\visual studio 2015\Projects\Bench.Test\bin\Debug\Bench.Test.vshost.exe.Config
LOG: Using host configuration file: 
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework\v4.0.30319\config\machine.config.
LOG: The same bind was seen before, and was failed with hr = 0x80070002.

[adamsitnik]

Thanks for the detailed info. I am afraid that currently I can offer you only some workarounds (choose one):

• Install msbuild 12
• Remove the references to msbuild 12 dlls that BenchmarkDotNet has added to your project and reference it's VS 215 counterparts (located probably at "C:\Program Files (x86)\MSBuild\14.0\Bin, instead of 14 they contain "Core" in it's names like Microsoft.Build.Tasks.Core.dll)
• Reterget your project to .NET 4.0 (4.0 version of BenchmarkDotNet uses other version of msbuild)

[ig-sinicyn]

@adamsitnik
Currently I've just reverted back to 0.9.3. I've tried already the second workaround as the one with the least footprint and it did not work. If you have working sample with references fixed, can you attach it? Thanks!

Any plans to fix the issue in the near future?

[adamsitnik]

Any plans to fix the issue in the near future?

Sure thing! The problem is that I am not sure if a perfect solution exists. I will get back to you soon.

Currently I've just reverted back to 0.9.3.

0.9.4 contains some nice export's perf improvements, so if you are running a lot of benchmarks I would recommend installing msbuild 12 (13?)

[ig-sinicyn]

I would recommend installing msbuild 12 (13?)

I wish I could:)

Any non-nuget prerequisite is a showstopper for most projects I've participated for.
The same is true if tests are runned on CI service you have no control on.

[adamsitnik]

I wish I could:)

I understand ;) I think that we should publish a new version of BenchmarkDotNet that has strict net46 target that references msbuild dlls from VS 2015. So then any net46+ project that either way must use VS 2015 and uses BDN should work.

[ig-sinicyn]

Ok, will wait for that!:)

Thanks one more time)

[adamsitnik]

@ig-sinicyn you welcome! The bug is fixed, I have pushed the changes to our develop branch.

@AndreyAkinshin @mattwarren Some background:

• We had some issues with compilation of .NET 4.5+ projects, so I upgraded dependencies to msbuild dlls to use v12 for our .NET 4.5 target as part of 0.9.4
• it turns out that people who do not have MSBuild 2013 installed do not have Microsoft.Build.Utilities.v12.0.dll in GAC and fail to use 0.9.4 with .NET 4.5+ target
• Now I added new NET46 target to BDN that references MSBuild 14 dlls to avoid that.

So now:

• if people target .NET40 (use VS 2010) they use Microsoft.Build.Utilities.v4.0
• if people target .NET45 (use VS 2013) they use Microsoft.Build.Utilities.v12.0
• if people target .NET46 (use VS 2015) they use Microsoft.Build.Utilities.Core

Is it a perfect solution? Not and I am not sure if perfect one exists. The problematic scenario is when people use VS 2015, do not have MSBuild 13 installed and target .NET 4.5 ;)

[AndreyAkinshin]

@adamsitnik, can we use our magic batch file (which try to detect an existed MSBuild version) as a fallback scenario in a case of fail?

[adamsitnik]

@AndreyAkinshin Yes, we can. I am going to add it, thanks for the hint. Probably we could also call msbuild like we do call dotnet cli (Process.Start("msbuild")). I will check this and come back with the results. Maybe this could be the best fix?

[AndreyAkinshin]

We can don't have msbuild in the path variable. I suggest to use managed API as a primary way, and use process.start (or batch file) as a second option.

[adamsitnik]

We can don't have msbuild in the path variable

What a pity :( It would simplify the code and project.json file ;)

[adamsitnik]

@AndreyAkinshin You were right, no MSBuild in PATH can happen (I checked that). I have added the fallback + to improve perf I generate the bat only when it is needed. You can find my changes at develop

[mattwarren]

@adamsitnik BTW the reason that BuildBenchmark.bat was always generated was because someone asked for it, see #40 (although the actual reason why isn't in the issue).

So unless the perf is terrible you might want to leave that functionality in or at least try and ping @constructor-igor about it.

[adamsitnik]

@mattwarren Thanks for pointing this. I will re-add this and generate bat for dnx/core as well.

[niemyjski]

I'm targeting .net 4.5.2 with a VS2013 image on appveyor (default) https://ci.appveyor.com/project/Exceptionless/exceptionless-net and I'm getting errors saying its trying to load Could not load file or assembly 'Microsoft.Build.Framework, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

[AndreyAkinshin]

Hey @jgrandyxignite. I guess, you are trying to run benchmark from unit tests, is it correct? In this case, you have to disable "shadow copy".