Benchmarkdotnet: Friendliness to LinqPad

[omariom]

Would be nice to have ability to run benchmarks in LinqPad and get nicely formattet results in LinqPad's Results output.

[AndreyAkinshin]

Good idea, added to roadmap: https://github.com/PerfDotNet/BenchmarkDotNet/wiki

[mattwarren]

We actually have LinqPad support (or we did, I haven't tested it since), I made it work as part of #42.

So maybe it's enough to just do something like this in the .linq file:

public static void Main()
{
    var report = new BenchmarkRunner().Run<Intro_00_Basic>();
    report.Dump();
}

[mattwarren]

@AndreyAkinshin sorry, I've only just read the note on the wiki:

Please, use the develop branch for developing. The master branch should correspond the latest NuGet package of the library.

I'll make sure I use the develop branch in the future.

[AndreyAkinshin]

@mattwarren, yes, we should implement the Dump method.

[mattwarren]

Sorry, I wasn't clear. Dump() is built-in to LinqPad and it (usually) does a pretty good job of displaying whatever object you give it.

When I get a chance I'll run that LinqPad code and post the output here.

[mattwarren]

We actually have LinqPad support (or we did, I haven't tested it since), I made it work as part of #42.

Whoops, seems like our LinqPad support broke somewhere along the way, I'll take a look at fix it.

[pjmagee]

Yep, it says it cannot find UserQuery namespace. :(

[mattwarren]

@adamsitnik looks like your recent changes fixed this also. When running against your references branch (i.e. PR #125) in LINQPad I get the following output (i.e. it works!):

// **************************
// Benchmark: Intro_00_Basic_Sleep_Dry
// *** Generate *** 
// File doesn't exist: C:\Program Files (x86)\LINQPad4\benchmarkdotnet.dll
//   Actually at: C:\Users\warma11\AppData\Local\Temp\LINQPad\_abcwrien\shadow_bljeos\benchmarkdotnet.dll
//   Copying benchmarkdotnet.dll
//   from: C:\Users\warma11\AppData\Local\Temp\LINQPad\_abcwrien\shadow_bljeos
//   to: C:\Program Files (x86)\LINQPad4
// File doesn't exist: C:\Program Files (x86)\LINQPad4\query_inrgje.dll
//   Actually at: C:\Users\warma11\AppData\Local\Temp\LINQPad\_abcwrien\query_inrgje.dll
//   Copying query_inrgje.dll
//   from: C:\Users\warma11\AppData\Local\Temp\LINQPad\_abcwrien
//   to: C:\Program Files (x86)\LINQPad4
// Result = Success
// DirectoryPath = C:\Program Files (x86)\LINQPad4\Intro_00_Basic_Sleep_Dry

// *** Build ***
// Result = Success

// *** Execute ***
// Launch: 1 / 1
...

I think we'll always need to fall-back to the File doesn't exist.. stuff (see EnsureDependancyInCorrectLocation), because of the way that LINQPad works, unless you have any ideas?

But even this workaround (EnsureDependancyInCorrectLocation) didn't always work before your changes, so I think you fixed something :-)

[adamsitnik]

@mattwarren I have just tried running simple benchmark from LINQPad 5 and it worked. But I could not attach debugger to see what is going on. However I have removed the workaround method calls and it worked too. I had to remove copy of benchmarkdotnet.dll from C:\Program Files (x86)\LINQPad5 to see the changes. Could you try it too? We should not have any missing dll problems now because we set absolute path. I suspect that relative paths and different CurrentDirectory were the cause of previous problems with LINQPad.

[mattwarren]

@adamsitnik I just tried the same thing (i.e. removing calls to EnsureDependancyInCorrectLocation) and you're right, since your changes we no longer seem to need the EnsureDependancyInCorrectLocation(..) code.

Are you happy to remove all the EnsureDependancyInCorrectLocation code as part of your PR?

[adamsitnik]

@mattwarren I just did some minor refactoring, as soon as test runner finishes I push it.

[adamsitnik]

@mattwarren btw I tested it for both LINQPad 4 & 5 and it works fine ;)

[AndreyAkinshin]

@adamsitnik, @mattwarren, I checked the last version in LINQPad, it works for me too. Everything looks fine.
@adamsitnik, I'm ready to accept the PR, waiting for your last commits.

[adamsitnik]

@AndreyAkinshin I have just pushed my last commit, minor refactor to separate things common for all generators

[omariom]

:+1:

[mattwarren]

@omariom I've also done a bit of work on getting BenchmarkDotNet to display result in LINQPad's GUI output, So far I've got this working:

Is this what you were after?

[omariom]

@mattwarren

yes, exactly :+1:

[centur]

Can I just bump this thread and check if anyone still can run BenchmarkDotNet v.0.9.9 with latest LinqPad v5.08.01(AnyCPU)?

I tried to use nuget from a public feed as well as from https://ci.appveyor.com/nuget/benchmarkdotnet. Both versions (0.9.9 and v0.9.9.123)I've tried don't work.

When running a sample from Getting started I got this output:

// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   Md5VsSha256_Sha256
//   Md5VsSha256_Md5

// Validating benchmarks:
Assembly query_qouryv, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null which defines benchmarks references non-optimized LINQPad
// **************************
// Benchmark: Md5VsSha256_Sha256
// *** Generate *** 
// Result = Success
// BinariesDirectoryPath = C:\Users\Alexey\AppData\Local\Temp\LINQPad5\_iudyussc

// *** Build ***
BuildScript: C:\Users\Alexey\AppData\Local\Temp\LINQPad5\_iudyussc\BDN.Auto.bat

and then MissingMethodException:
Method not found: 'System.Collections.Immutable.ImmutableArray1 System.Reflection.Metadata.MetadataReader.GetBlobContent(System.Reflection.Metadata.BlobHandle)'.
`

With a StackTrace of

   at Microsoft.CodeAnalysis.MetadataReaderExtensions.CreateAssemblyIdentityOrThrow(MetadataReader reader, Version version, AssemblyFlags flags, BlobHandle publicKey, StringHandle name, StringHandle culture, Boolean isReference)
   at Microsoft.CodeAnalysis.MetadataReaderExtensions.ReadAssemblyIdentityOrThrow(MetadataReader reader)
   at Microsoft.CodeAnalysis.PEAssembly..ctor(AssemblyMetadata owner, ImmutableArray`1 modules)
   at Microsoft.CodeAnalysis.AssemblyMetadata.GetOrCreateData()
   at Microsoft.CodeAnalysis.AssemblyMetadata.GetModules()
   at Microsoft.CodeAnalysis.AssemblyMetadata.IsValidAssembly()
   at Microsoft.CodeAnalysis.CommonReferenceManager`2.GetMetadata(PortableExecutableReference peReference, CommonMessageProvider messageProvider, Location location, DiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CommonReferenceManager`2.ResolveMetadataReferences(TCompilation compilation, Dictionary`2 assemblyReferencesBySimpleName, ImmutableArray`1& references, IDictionary`2& boundReferenceDirectiveMap, ImmutableArray`1& boundReferenceDirectives, ImmutableArray`1& assemblies, ImmutableArray`1& modules, DiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.ReferenceManager.CreateAndSetSourceAssemblyFullBind(CSharpCompilation compilation)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.ReferenceManager.CreateSourceAssemblyForCompilation(CSharpCompilation compilation)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetRuntimeMetadataVersion(EmitOptions emitOptions)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.GetRuntimeMetadataVersion(EmitOptions emitOptions, DiagnosticBag diagnostics)
   at Microsoft.CodeAnalysis.CSharp.CSharpCompilation.CreateModuleBuilder(EmitOptions emitOptions, IMethodSymbol debugEntryPoint, IEnumerable`1 manifestResources, CompilationTestData testData, DiagnosticBag diagnostics, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(EmitStreamProvider peStreamProvider, EmitStreamProvider pdbStreamProvider, EmitStreamProvider xmlDocumentationStreamProvider, EmitStreamProvider win32ResourcesStreamProvider, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, CompilationTestData testData, Func`1 getHostDiagnostics, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, CompilationTestData testData, Func`1 getHostDiagnostics, CancellationToken cancellationToken)
   at Microsoft.CodeAnalysis.Compilation.Emit(Stream peStream, Stream pdbStream, Stream xmlDocumentationStream, Stream win32Resources, IEnumerable`1 manifestResources, EmitOptions options, IMethodSymbol debugEntryPoint, CancellationToken cancellationToken)
   at BenchmarkDotNet.Toolchains.Classic.RoslynBuilder.Build(GenerateResult generateResult, ILogger logger, Benchmark benchmark)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Build(ILogger logger, IToolchain toolchain, GenerateResult generateResult, Benchmark benchmark)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark[] benchmarks, ILogger logger, String title, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark[] benchmarks, IConfig config, Func`2 toolchainProvider)
   at UserQuery.Main() in C:\Users\Alexey\AppData\Local\Temp\LINQPad5\_iudyussc\query_jspqpw.cs:line 49
   at LINQPad.ExecutionModel.ClrQueryRunner.Run()
   at LINQPad.ExecutionModel.Server.RunQuery(QueryRunner runner)
   at LINQPad.ExecutionModel.Server.StartQuery(QueryRunner runner)
   at LINQPad.ExecutionModel.Server.<>c__DisplayClass152_0.<ExecuteClrQuery>b__0()
   at LINQPad.ExecutionModel.Server.SingleThreadExecuter.Work()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart() 

Any hints of what I might do wrong are really appreciated

[Reonekot]

@centur - did you figure it out ? I'm having the same issues with newest official + beta of LINQPad5. Tried both AnyCPU and x86 version and from CMD using "lprun -optimize".

When running I just get at the top:

// Validating benchmarks:
Assembly query_kzwwnu, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null which defines benchmarks references non-optimized LINQPad

And at the end:

// * Detailed results *
Md5VsSha256_Sha256
<Empty statistic (N=0)>

Md5VsSha256_Md5
<Empty statistic (N=0)>

Total time: 00:00:07 (7.62 sec)

// * Summary *

Host Process Environment Information:
BenchmarkDotNet.Core=v0.9.9.0
OS=Microsoft Windows NT 6.2.9200.0
Processor=Intel(R) Xeon(R) CPU E3-1240 v5 3.50GHz, ProcessorCount=8
Frequency=3421873 ticks, Resolution=292.2376 ns, Timer=TSC
CLR=MS.NET 4.0.30319.42000, Arch=32-bit ?
GC=Concurrent Workstation
JitModules=clrjit-v4.6.1586.0

Type=Md5VsSha256  Mode=Throughput

 Method | Median | StdDev |
------- |------- |------- |
 Sha256 |     NA |     NA |
    Md5 |     NA |     NA |

Benchmarks with issues:
  Md5VsSha256_Sha256
  Md5VsSha256_Md5

// ***** BenchmarkRunner: End *****

[centur]

@Reonekot Nope, using their sample console app :(

[adamsitnik]

I will take a look at this, but if the error is:

and then MissingMethodException:
Method not found: 'System.Collections.Immutable.ImmutableArray1 System.Reflection.Metadata.MetadataReader.GetBlobContent(System.Reflection.Metadata.BlobHandle)'.

then it means that we have some dll version issue. Most probably LINQPad is using older version of the dll, when we try to compile the Benchmarks we get dll which is already loaded but it does not contain given method.

[Reonekot]

I have seen the same exception from within LINQPad (MissingMethodException) twice, but not on most runs and never when running with lprun.exe. With lprun it just gives the output I posted above - no exceptions or other signs of errors, besides it seems it just skips running the tests.

[Reonekot]

Looks like the error has change in the new versions for Benchmark.net - seems like LINQPad does something "wrong" since it tries to resolve the wrong .dll? (.Core)

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'BenchmarkDotNet.Core, Version=0.10.0.0, Culture=neutral, PublicKeyToken=aa0ca2f9092cefc4' or one of its dependencies. The system cannot find the file specified.
at BenchmarkDotNet.Autogenerated.Runnable.Run(Boolean isDiagnoserAttached)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at BenchmarkDotNet.Autogenerated.Program.Main(String[] args)

[ipatalas]

Anyone can successfully run BDN in LINQPad?

I'm using BDN 0.10.4 and LINQPad 5.22.02 and getting the following results:

Command line used was lprun -optimize <filename>

[adamsitnik]

With help from @albahari we were able to bring back LINQPad support!

You need 5.22.05+ build of LINQPad available here

Once you have it you need to install our 0.10.6.185 package from our custom CI feed available at https://ci.appveyor.com/nuget/benchmarkdotnet.

Reminder: remember to use AnyCPU build of LINQPad so you will be able to benchmark both 32 and 64 bit. And don't forget to turn ON the optimizations.