Benchmarkdotnet: NativeMemoryProfiler doesn't report allocations in v0.12.1.1432 Nightly

[JimBobSquarePants]

I'm running the following benchmark to compare graphics libraries.

https://github.com/JimBobSquarePants/core-imaging-playground/tree/ac506f591f7448cb4b2250c7fb9717a976faf198

With v0.12.1 I get the following result:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.572 (2004/?/20H1)
Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.403
  [Host]            : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  .Net Core 3.1 CLI : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT

Job=.Net Core 3.1 CLI  Toolchain=.NET Core 3.1  IterationCount=5
LaunchCount=1  WarmupCount=5

| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
|-------------------------------------- |----------:|----------:|---------:|------:|----------:|------:|------:|-----------:|------------------------:|-------------------:|
| 'System.Drawing Load, Resize, Save' | 396.97 ms | 14.314 ms | 2.215 ms | 1.00 | - | - | - | 10.2 KB | 11375.79 KB | 738.27 KB |
| 'ImageSharp Load, Resize, Save' | 221.30 ms | 6.784 ms | 1.762 ms | 0.56 | - | - | - | 1985.19 KB | 24.9 KB | 9.14 KB |
| 'ImageMagick Load, Resize, Save' | 437.39 ms | 17.609 ms | 4.573 ms | 1.10 | - | - | - | 53.89 KB | 60956.9 KB | 0.66 KB |
| 'ImageFree Load, Resize, Save' | 235.38 ms | 3.423 ms | 0.889 ms | 0.59 | 6000.0000 | - | - | 69.75 KB | 49140.58 KB | 58.91 KB |
| 'MagicScaler Load, Resize, Save' | 72.62 ms | 1.089 ms | 0.168 ms | 0.18 | - | - | - | 103.91 KB | 3277.21 KB | 20.98 KB |
| 'SkiaSharp Canvas Load, Resize, Save' | 251.91 ms | 8.000 ms | 2.078 ms | 0.63 | - | - | - | 99.39 KB | 69715.49 KB | 1663.01 KB |
| 'SkiaSharp Bitmap Load, Resize, Save' | 251.14 ms | 5.455 ms | 1.417 ms | 0.63 | - | - | - | 85.47 KB | 70425.35 KB | 494.62 KB |
| 'NetVips Load, Resize, Save' | 150.29 ms | 2.230 ms | 0.579 ms | 0.38 | - | - | - | 47.14 KB | 84730.41 KB | 7532.25 KB |

However, I thought the reported leaks seems odd. I saw #1427 and the merged fix in #1451 so updated to the latest (at this time v0.12.1.1432-nightly) to take advantage of the fix.

Unfortunately all native allocations now go unreported:

BenchmarkDotNet=v0.12.1.1432-nightly, OS=Windows 10.0.19041.572 (2004/May2020Update/20H1)
Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.403
  [Host]            : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT
  .Net Core 3.1 CLI : .NET Core 3.1.9 (CoreCLR 4.700.20.47201, CoreFX 4.700.20.47203), X64 RyuJIT

Job=.Net Core 3.1 CLI  Toolchain=.NET Core 3.1  IterationCount=5
LaunchCount=1  WarmupCount=5

| Method | Mean | Error | StdDev | Ratio | Gen 0 | Gen 1 | Gen 2 | Allocated | Allocated native memory | Native memory leak |
|-------------------------------------- |----------:|----------:|---------:|------:|----------:|------:|------:|-----------:|------------------------:|-------------------:|
| 'System.Drawing Load, Resize, Save' | 402.30 ms | 25.398 ms | 6.596 ms | 1.00 | - | - | - | 10.2 KB | - | - |
| 'ImageSharp Load, Resize, Save' | 221.72 ms | 17.159 ms | 4.456 ms | 0.55 | - | - | - | 1985.19 KB | - | - |
| 'ImageMagick Load, Resize, Save' | 429.75 ms | 21.762 ms | 5.651 ms | 1.07 | - | - | - | 53.89 KB | - | - |
| 'ImageFree Load, Resize, Save' | 234.83 ms | 8.620 ms | 2.239 ms | 0.58 | 6000.0000 | - | - | 69.73 KB | - | - |
| 'MagicScaler Load, Resize, Save' | 73.62 ms | 4.423 ms | 1.149 ms | 0.18 | - | - | - | 102.78 KB | - | - |
| 'SkiaSharp Canvas Load, Resize, Save' | 252.89 ms | 14.303 ms | 2.213 ms | 0.63 | - | - | - | 99.34 KB | - | - |
| 'SkiaSharp Bitmap Load, Resize, Save' | 250.36 ms | 4.874 ms | 0.754 ms | 0.62 | - | - | - | 85.47 KB | - | - |
| 'NetVips Load, Resize, Save' | 148.82 ms | 2.542 ms | 0.393 ms | 0.37 | - | - | - | 47.14 KB | - | - |

I was going to upload the traces but they are far too big for my bandwidth to allow. Please let me know if you require any more information.

[adamsitnik]

@JimBobSquarePants thank you for a very detailed bug report!

@WojciechNagorski could you please take a look?

[WojciechNagorski]

I want to fix it, I'll have time in a week or two.

I think that I remove too much after the last review:

https://github.com/dotnet/BenchmarkDotNet/pull/1451/commits/5a241e02a8b86da040c293f9bf1f6f55fa5f9414#diff-abccc6ab9121978fe6440972d6e3fdc82f0d4b85d91b84f149ea6f10198633faL39-R41

[adamsitnik]

@JimBobSquarePants when this build gets green (around 30-40 minutes from now) our CI should produce a new NuGet package with version 0.12.1.1467. Could you please give it a try and let us know if it works fine?

[JimBobSquarePants]

@adamsitnik Sorry just saw this!

Looks like its's working. I'm getting readings in my benchmark and ImageSharp is no longer reporting native memory leaks which is expected.

[WojciechNagorski]

@JimBobSquarePants Thanks for the information! If you find other problems, don't hesitate to create a new issue.

[adamsitnik]

@JimBobSquarePants thank you!