Benchmarkdotnet: How to get benchmarks running from LINQPad?

[michael-wolfenden]

Running the following code in LINQPad5 v5.25.00 referencing BenchmarkDotNet 0.10.10.339

void Main()
{
    var summary = BenchmarkRunner.Run<Md5VsSha256>();
}

public class Md5VsSha256
{
    private const int N = 10000;
    private readonly byte[] data;

    private readonly SHA256 sha256 = SHA256.Create();
    private readonly MD5 md5 = MD5.Create();

    public Md5VsSha256()
    {
        data = new byte[N];
        new Random(42).NextBytes(data);
    }

    [Benchmark]
    public byte[] Sha256() => sha256.ComputeHash(data);

    [Benchmark]
    public byte[] Md5() => md5.ComputeHash(data);
}

Outputs the following and does not run

// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   Md5VsSha256.Sha256: DefaultJob
//   Md5VsSha256.Md5: DefaultJob

// Validating benchmarks:
Assembly LINQPadQuery which defines benchmarks references non-optimized LINQPad
Assembly LINQPadQuery which defines benchmarks is non-optimized
Benchmark was built without optimization enabled (most probably a DEBUG configuration). Please, build it in RELEASE.
Assembly LINQPadQuery, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null is located in temp. If you are running benchmarks from xUnit you need to disable shadow copy. It's not supported by design.
// * Artifacts cleanup *

This appears to be a regression as the same script referencing BenchmarkDotNet 0.10.8.246 outputs

// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   Md5VsSha256.Sha256: DefaultJob
//   Md5VsSha256.Md5: DefaultJob

// Validating benchmarks:
Assembly LINQPadQuery which defines benchmarks references non-optimized LINQPad
Assembly LINQPadQuery which defines benchmarks is non-optimized
// **************************
// Benchmark: Md5VsSha256.Sha256: DefaultJob
// *** Generate *** 
// Result = Success
// BinariesDirectoryPath = C:\Users\WOLFENDEN-PC\AppData\Local\Temp\LINQPad5\_vhnggtqv\txnrcv

// *** Build ***
BuildScript: C:\Users\WOLFENDEN-PC\AppData\Local\Temp\LINQPad5\_vhnggtqv\txnrcv\67c72de2-c073-4a54-9515-f24267bef6e7.bat
// Result = Success

// *** Execute ***
// Launch: 1 / 1
// Execute: C:\Users\WOLFENDEN-PC\AppData\Local\Temp\LINQPad5\_vhnggtqv\txnrcv\67c72de2-c073-4a54-9515-f24267bef6e7.exe 
.....
.....
.....

[AndreyAkinshin]

It's not a bug, it's a feature. We have decided that BenchmarkDotNet shouldn't allow running benchmarks in DEBUG. Why do you want to run your benchmark without optimizations?
See also: https://github.com/dotnet/BenchmarkDotNet/issues/561
/cc @adamsitnik

[michael-wolfenden]

So I have set the "compile with /optimize+" setting in linqpad, which from my understanding is the equivalent of release mode, and it still doesn't run

// ***** BenchmarkRunner: Start   *****
// Found benchmarks:
//   Md5VsSha256.Sha256: DefaultJob
//   Md5VsSha256.Md5: DefaultJob

// Validating benchmarks:
Assembly LINQPadQuery which defines benchmarks references non-optimized LINQPad
Assembly LINQPadQuery, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null is located in temp. If you are running benchmarks from xUnit you need to disable shadow copy. It's not supported by design.
// * Artifacts cleanup *

Or am I misunderstanding and the issue is that linqpad itself is not optimized?

[adamsitnik]

Hello @michael-wolfenden

Thanks for reporting your issue. I have just pushed a commit with the fix (we detect that user is using LINQPad and has optimizations disabled and give nice error message).

However as you have noticed LINQPad itself is non-optimized. To overcome this problem please use custom config with our policy disabled:

public class AllowNonOptimized : ManualConfig
{
    public AllowNonOptimized()
    {
        Add(JitOptimizationsValidator.DontFailOnError); // ALLOW NON-OPTIMIZED DLLS

        Add(DefaultConfig.Instance.GetLoggers().ToArray()); // manual config has no loggers by default
        Add(DefaultConfig.Instance.GetExporters().ToArray()); // manual config has no exporters by default
        Add(DefaultConfig.Instance.GetColumnProviders().ToArray()); // manual config has no columns by default
    }
}

BenchmarkRunner<YourType>(new AllowNonOptimized());

[adamsitnik]

@albahari is there any reason why LINQPad is non-optimized? It would make our user experience much better if it was optimized. You can use following settings in your csproj to have nice symbol information in pdb and optimizations enabled for Release:

<DebugType>pdbonly</DebugType>
<DebugSymbols>true</DebugSymbols>

[albahari]

Because the performance-critical libraries that LINQPad uses are optimized (libraries such as Roslyn & ActiPro's syntax editor), there's not much to gain in optimizing LINQPad itself. And there's a lot to lose - namely, the quality of the diagnostic information when an error occurs. Just recently, I've diagnosed an intermittent hang based on a stack trace which might have been compromised with optimizations enabled. I think users would rather have better reliability and quicker bug fixes than the small perf gain that optimizations would yield in this situation.

In the case of BenchmarkDotNet, the message about LINQPad not being optimized is a false alarm, because LINQPad's non-optimized code should not find its way into the code paths being tested (unless you're performance-testing features of LINQPad itself - such as LINQPad's Dump or Dif methods). LINQPad is like the IDE, so it shouldn't really be counted in this scenario.

Could I can add an assembly attribute to LINQPad to signal this? Obviously I can't take a dependency on BenchmarkDotNet, but I could define an attribute class like the following in LINQPad, and you could detect it by looking at the full type name:

namespace BenchmarkDotNet {
   [Serializable, AttributeUsageAttribute (AttributeTargets.Assembly)]
   class AllowNonoptimized : System.Attribute { }
}

[adamsitnik]

@albahari thanks for the explanation!

I have added LINQPad to our "white list", so everything should work fine now.

[ZodmanPerth]

@adamsitnik Will this require a new release of BenchmarkDotNet or has it already made it into a release? I've just moved to a new device and had to employ the custom config workaround (which in my case is not a big deal; this is more of a notification type question).

[adamsitnik]

@ZodmanPerth hi! It will be a part of 0.10.11.

@AndreyAkinshin could you trigger our CI build? It has failed for some magical reason

@ZodmanPerth as soon as our CI builds the package, it should be available at

<packageSources>
  <add key="bdn-nightly" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>

[AndreyAkinshin]

@adamsitnik, done.

[ZodmanPerth]

Thanks.
There was a minor update at that address (0.10.10.343) dated November 18. It doesn't appear to be the build I'm after, but just in case here's part of the report I get:
Assembly LINQPadQuery which defines benchmarks references non-optimized LINQPad If you own this dependency, please, build it in RELEASE. If you don't, you can create custom config with DontFailOnError to disable our custom policy and allow this benchmark to run. Assembly LINQPadQuery, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null is located in temp. If you are running benchmarks from xUnit you need to disable shadow copy. It's not supported by design.
I'll check back later for 0.10.11.

[adamsitnik]

Hi @ZodmanPerth

Our CI had some issues. Now we publish NuGet package again ;) You can try 0.10.10.350 from our CI feed:

<packageSources>
  <add key="bdn-nightly" value="https://ci.appveyor.com/nuget/benchmarkdotnet" />
</packageSources>

[ZodmanPerth]

Thanks @adamsitnik . By the time I was able to take action on this the nightly build was up to 0.10.10.353, which I confirm does have the fix in it. Good job!

[adamsitnik]

@ZodmanPerth thanks for the confirmation!