Benchmarkdotnet: Benchmark fails when used in conjunction with the new Microsoft.AspNetCore.Mvc.Testing (.NET Core 2.1)

[mauroservienti]

We're in the need of running benchmarks end-to-end for a MVC Core App. However benchmarks should not be affected by HTTP processing, we just need to exercise the entire MVC pipeline, from routing down to the controller and back, with our middlewares included.

With the release of .NET Core 2.1, Microsoft.AspNetCore.Mvc.Testing has been released as well, allowing something similar to:

var client = new WebApplicationFactory<MyApp.Startup>()
                .CreateClient(new WebApplicationFactoryClientOptions
                {
                    AllowAutoRedirect = false
                });
await client.GetAsync("some/valid/url");

We then tried to apply a benchmark to the above code, but when executed it fails with the following exception:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.InvalidOperationException: Can't find'C:\dev\mauroservienti\BenchmarkDotNetAspNetCoreIssueRepro\BenchmarkDotNetAspNetCoreIssueRepro\bin\Release\netcoreapp2.1\62bb9eb0-90ed-4543-8db5-d6b24cf72c77\bin\Release\netcoreapp2.1\AspNetCoreApi.deps.json'. This file is required for functional tests to run properly. There should be a copy of the file on your source project bin folder. If that is not the case, make sure that the property PreserveCompilationContext is set to true on your project file. E.g 'true'. For functional tests to work they need to either run from the build output folder or the AspNetCoreApi.deps.json file from your application's output directory must be copied to the folder where the tests are running on. A common cause for this error is having shadow copying enabled when the tests run.
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory1.EnsureDepsFile() at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory1.EnsureServer()
at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory1.CreateDefaultClient(DelegatingHandler[] handlers) at Microsoft.AspNetCore.Mvc.Testing.WebApplicationFactory1.CreateClient(WebApplicationFactoryClientOptions options)
at BenchmarkDotNetAspNetCoreIssueRepro.Tests.Setup() in C:\dev\mauroservienti\BenchmarkDotNetAspNetCoreIssueRepro\BenchmarkDotNetAspNetCoreIssueRepro\Program.cs:line 27
at BenchmarkDotNet.Autogenerated.Runnable.Run(IHost host) in C:\dev\mauroservienti\BenchmarkDotNetAspNetCoreIssueRepro\BenchmarkDotNetAspNetCoreIssueRepro\bin\Release\netcoreapp2.1\62bb9eb0-90ed-4543-8db5-d6b24cf72c77\62bb9eb0-90ed-4543-8db5-d6b24cf72c77.notcs:line 122
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in C:\dev\mauroservienti\BenchmarkDotNetAspNetCoreIssueRepro\BenchmarkDotNetAspNetCoreIssueRepro\bin\Release\netcoreapp2.1\62bb9eb0-90ed-4543-8db5-d6b24cf72c77\62bb9eb0-90ed-4543-8db5-d6b24cf72c77.notcs:line 55

Clearly WebApplicationFactory is not happy because of the missing .deps.json file, that is not copied by BenchmarkDotNet (as far as I can tell) to the location where benchmarks are executed.

A repro is available here: https://github.com/mauroservienti/BenchmarkDotNetAspNetCoreIssueRepro

[adamsitnik]

Hi @mauroservienti

I have forked your repo and reproduced the problem, however, failed to solve it.

I was expecting that setting the current directory to the web app location will be enough to make Microsoft.AspNetCore.Mvc.Testing work.

I tried;

Directory.SetCurrentDirectory(Path.GetDirectoryName(typeof(AspNetCoreApi.Startup).Assembly.Location));

and

.WithWebHostBuilder(config => config.ConfigureAppConfiguration((ctx, cb) => ctx.HostingEnvironment.ContentRootPath = Path.GetDirectoryName(typeof(AspNetCoreApi.Startup).Assembly.Location)))

But it did not help.

I am afraid that as of today it's not supported scenario.

[mauroservienti]

thanks for the feedback @adamsitnik

[stukselbax]

Will you plan to support this scenario? Faced the same problem when tried to benchmark http endpoint.

[adamsitnik]

Most probably not, BenchmarkDotNet was designed for microbenchmarks, not web apps.

ASP.NET Team has built a nice set of tools for benchmarking ASP.NET Core apps, you should give it a try: https://github.com/aspnet/Benchmarks/

[martincostello]

I've been able to do this by referencing the relevant ASP.NET Core dependencies manually in the BenchmarkDotNet project, rather than relying on them to be picked up transitively, as well as including a reference to Microsoft.AspNetCore.Mvc.Testing.

[stukselbax]

@mauroservienti Failed to use your proposed solution. Have tried:

  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.11.5" />
    <PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Core" Version="2.2.5" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="2.2.0" />
  </ItemGroup>

with no luck. The only solution that works is copying (yet manually) file Assembly.deps.json to the required directory, which is generated when benchmarks run. But this is annoying.

[martincostello]

Here's a snippet of our working one:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFrameworks>netcoreapp2.2</TargetFrameworks>
  </PropertyGroup>
  <ItemGroup>
    <WebApplicationFactoryContentRootAttribute Include="MyApp" AssemblyName="MyApp" ContentRootPath="$([System.IO.Path]::GetFullPath('$(MSBuildThisFileDirectory)\..\..\src\MyApp'))" ContentRootTest="MyApp.csproj" Priority="-1" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\..\src\MyApp\MyApp.csproj" />
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.11.5" />
    <PackageReference Include="Microsoft.AspNetCore" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Authentication.Cookies" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Hosting" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Formatters.Json" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.NodeServices" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Razor" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.Razor.Runtime" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.ResponseCaching" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.ResponseCompression" Version="2.2.0" />
    <PackageReference Include="Microsoft.AspNetCore.StaticFiles" Version="2.2.0" />
    <PackageReference Include="Microsoft.Extensions.Http" Version="2.2.0" />
  </ItemGroup>
</Project>

[sleemer]

@adamsitnik

I belive that whenever to use BenchmarkDotNet for testing web apps is more philosophical than practical question. Using it for testing controllers, imo, should be absolutely fine. And one way to solve the problem described by @mauroservienti would be using targets to copy *.deps.json files to the output folder. Similar to what Microsoft.AspNetCore.Mvc.Testing does by itself (There is a copy task in Microsoft.AspNetCore.Mvc.Testing.targets, that is responsible for copying all *.deps.json files).

[samueldjack]

I was able to get this scenario working by doing two things:

1. Adding the following to the csproj file for my benchmark exe to ensure that the deps.json file gets copied in:

   <!--
     Work around https://github.com/NuGet/Home/issues/4412. MVC uses DependencyContext.Load() which looks next to a .dll
     for a .deps.json.
   -->
   <Target Name="CopyDepsFiles" AfterTargets="Build">
     <ItemGroup>
       <DepsFilePaths Include="$([System.IO.Path]::ChangeExtension('%(_ResolvedProjectReferencePaths.FullPath)', '.deps.json'))" />
     </ItemGroup>
   
     <Copy SourceFiles="%(DepsFilePaths.FullPath)" DestinationFolder="$(OutputPath)" Condition="Exists('%(DepsFilePaths.FullPath)')" />
   </Target>
   

2. Use the InProcess job runner:

     BenchmarkRunner.Run<BenchmarkPageRender>(
           DefaultConfig
               .Instance
               .With(Job.InProcess.WithGcServer(true))
       );