Benchmarkdotnet: Question/Discussion: Benchmarking Unsafe Code

[Mike-E-angelo]

Hello Community,

(FWIW, I did do a search on "unsafe" in this repo and it did not return anything obvious so I am asking this question here.)

I recently got a nice pointer by @mattwarren on the Twitters to check out this really nice series that explores some serious guts of the .NET system.

In doing so, I managed to make a quick example of a 64-bit version of the UnsafeList featured in the series thanks to the help of this subsequent Twitter thread.

When running this code via the traditional eyeball-it-by-Stopwatch, this unsafe/native approach does seem faster, but when I run/measure it via Benchmark.NET, it reports that the unsafe code is about 15x slower. (Please note that I am strictly interested in writing as it would seem that the reading is ultimately left to an array access/read.)

So I am wondering, is there a setting/configuration with Benchmark.NET when it is measuring unsafe code? Is this a known issue? Is there something else here that might be a consideration? I am basically looking to square my understanding of the discrepancies here.

(BTW, is it considered blasphemy here to even mention Stopwatch in this repo? 😆)

Thank you for any context and assistance that you can provide. 👍

[AndreyAkinshin]

Hey @Mike-EEE, sorry for the late reply.

I guess, you have a problem with your config. Currently, you use:

ManualConfig.Create(DefaultConfig.Instance)
  .With(Job.Default.WithUnrollFactor(1).WithInvocationCount(1))

It forces BenchmarkDotNet to produce very short iterations. After the summary table, you should see the following notifications:

// * Warnings *
MinIterationTime
  Benchmarks.AssignArray: InvocationCount=1, UnrollFactor=1      -> MinIterationTime = 4.0000 us which is very small. It's recommended to increase it.
  Benchmarks.SumArray: InvocationCount=1, UnrollFactor=1         -> MinIterationTime = 5.0000 us which is very small. It's recommended to increase it.
  Benchmarks.AssignUnsafeList: InvocationCount=1, UnrollFactor=1 -> MinIterationTime = 41.0000 us which is very small. It's recommended to increase it.
  Benchmarks.SumUnsafeList: InvocationCount=1, UnrollFactor=1    -> MinIterationTime = 6.0000 us which is very small. It's recommended to increase it.

Such measurements are unreliable and meaningless; you can't use it for any kinds of conclusions. I tried to remove the invocation count requirement and get the following exception for AssignUnsafeList:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at BenchmarkDotNet.UnsafeListx64.UnsafeList`1.Add(T item) in /Users/akinshin/Sandbox/GitHub/BenchmarkDotNet.UnsafeListx64/BenchmarkDotNet.UnsafeListx64/UnsafeList.cs:line 26
   at BenchmarkDotNet.UnsafeListx64.Benchmarks.AssignUnsafeList() in /Users/akinshin/Sandbox/GitHub/BenchmarkDotNet.UnsafeListx64/BenchmarkDotNet.UnsafeListx64/Benchmarks.cs:line 68
   at BenchmarkDotNet.Autogenerated.Runnable.MainMultiAction(Int64 invokeCount) in /Users/akinshin/Sandbox/GitHub/BenchmarkDotNet.UnsafeListx64/BenchmarkDotNet.UnsafeListx64/bin/Release/netcoreapp2.1/d82a016c-f5a9-41ab-8886-1c0177681563/d82a016c-f5a9-41ab-8886-1c0177681563.notcs:line 620
   at BenchmarkDotNet.Engines.Engine.Jitting()
   at BenchmarkDotNet.Autogenerated.Runnable.Run(IHost host) in /Users/akinshin/Sandbox/GitHub/BenchmarkDotNet.UnsafeListx64/BenchmarkDotNet.UnsafeListx64/bin/Release/netcoreapp2.1/d82a016c-f5a9-41ab-8886-1c0177681563/d82a016c-f5a9-41ab-8886-1c0177681563.notcs:line 127
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at BenchmarkDotNet.Autogenerated.UniqueProgramName.AfterAssemblyLoadingAttached(String[] args) in /Users/akinshin/Sandbox/GitHub/BenchmarkDotNet.UnsafeListx64/BenchmarkDotNet.UnsafeListx64/bin/Release/netcoreapp2.1/d82a016c-f5a9-41ab-8886-1c0177681563/d82a016c-f5a9-41ab-8886-1c0177681563.notcs:line 55

BenchmarkDotNet assumes that your methods don't have any side effects. It's not true for this one:

[Benchmark]
public void AssignUnsafeList()
{
    for (var i = 0; i < _source.Length; i++)
    {
        _unsafe.Add(_source[i]);
    }
}

BenchmarkDotNet should be able to run the method again and again to achieve the good precision level. Your method has side effects (new elements in _unsafe), so it can't be used this way.
The problem here is not about unsafe code, it's about benchmark design. You have to find another way to evaluate the performance of UnsafeList.

[Mike-E-angelo]

OK @AndreyAkinshin thank you for taking the time to point that out. I did indeed have the bad configuration due to the errors that you encountered as well. The side effects makes perfect sense as well. Thank you for taking the time to check this out and explain! It is much appreciated. Closing this issue for now as I considered it answered.