Benchmarkdotnet: System.ArgumentException Thrown on Benchmark Validate.

[JimBobSquarePants]

There's a few too many details to note here but I've linked the exact source code timestamp which throws the exception. If you need more information please let me know.

Basically everything runs until the benchmark is to be validated and then the application throws.

https://github.com/SixLabors/ZlibStream/blob/233f7e57310dba98ad8ba19c2e610e65646186cf/tests/ZlibStream.Benchmarks/ZlibDeflateBenchmark.cs

You'll likely need to update the git submodule in the repository to get it to build. The following command should do it.

git submodule -q update --init --recursive

`bash Unhandled exception. System.ArgumentException: An item with the same key has already been added. Key: zlib.managed, Version=1.1.4.0, Culture=neutral, PublicKeyToken=0371adb4a80cedf9 at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior) at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value) at System.Runtime.InteropServices.GitInformation..ctor(String headdesc, String commit, String branchname, Assembly assembly) at System.Runtime.InteropServices.GitInformationAttribute..ctor(String headdesc, String commit, String branchname, Type assemblyType) at System.Reflection.CustomAttribute._CreateCaObject(RuntimeModule pModule, RuntimeType type, IRuntimeMethodInfo pCtor, Byte** ppBlob, Byte* pEndBlob, Int32* pcNamedArgs) at System.Reflection.CustomAttribute.CreateCaObject(RuntimeModule module, RuntimeType type, IRuntimeMethodInfo ctor, IntPtr& blob, IntPtr blobEnd, Int32& namedArgs) at System.Reflection.CustomAttribute.AddCustomAttributes(ListBuilder`1& attributes, RuntimeModule decoratedModule, Int32 decoratedMetadataToken, RuntimeType attributeFilterType, Boolean mustBeInheritable, ListBuilder`1 derivedAttributes) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeModule decoratedModule, Int32 decoratedMetadataToken, Int32 pcaCount, RuntimeType attributeFilterType) at System.Reflection.CustomAttribute.GetCustomAttributes(RuntimeAssembly assembly, RuntimeType caType) at System.Reflection.RuntimeAssembly.GetCustomAttributes(Type attributeType, Boolean inherit) at System.Attribute.GetCustomAttributes(Assembly element, Boolean inherit) at BenchmarkDotNet.Validators.JitOptimizationsValidator.Validate(ValidationParameters validationParameters)+MoveNext() at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext() at System.Linq.Enumerable.DistinctIterator`1.MoveNext() at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection) at BenchmarkDotNet.Running.BenchmarkRunnerClean.Validate(BenchmarkRunInfo[] benchmarks, ILogger logger) at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo benchmarkRunInfo, Dictionary`2 buildResults, IResolver resolver, ILogger logger, List`1 artifactsToCleanup, String resultsFolderPath, String logFilePath, StartedClock& runChronometer) at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos) at BenchmarkDotNet.Running.BenchmarkSwitcher.RunWithDirtyAssemblyResolveHelper(String[] args, IConfig config) at BenchmarkDotNet.Running.BenchmarkSwitcher.Run(String[] args, IConfig config) at ZlibStream.Benchmarks.Program.Main(String[] args) in C:\development\github\SixLabors\ZlibStream\tests\ZlibStream.Benchmarks\Program.cs:line 14

[adamsitnik]

Hello @JimBobSquarePants

This exception is very strange because BDN just calls:

https://github.com/dotnet/BenchmarkDotNet/blob/0b83c934e5f983ff2aa43c82e63d242216d330bf/src/BenchmarkDotNet/Extensions/AssemblyExtensions.cs#L17-L20

and it's the System.Attribute.GetCustomAttributes that calls the throwing code.

Is it possible that zlib.managed, Version=1.1.4.0 has the [DebuggableAttribute] defined twice?

You can check that by opening the file in ILSpy:

[JimBobSquarePants]

Thanks for investigating @adamsitnik The source code for that library is available so I will have a look and get back to you.

[JimBobSquarePants]

Everything looks normal. I'm confused.

[adamsitnik]

@JimBobSquarePants it can also be a bug in .NET itself. Which version of the framework are you using?

edit: It would be great if you could provide the output of the 2 following commands:

dotnet --info

Executed in the project with benchmarks:

dotnet run --info

[JimBobSquarePants]

@adamsitnik No worries, her you go...

dotnet --info

.NET Core SDK (reflecting any global.json):
 Version:   3.1.300-preview-015095
 Commit:    85c45057dc

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.18363
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\3.1.300-preview-015095\

Host (useful for support):
  Version: 3.1.3
  Commit:  4a9f85e9f8

.NET Core SDKs installed:
  1.1.14 [C:\Program Files\dotnet\sdk]
  2.1.700 [C:\Program Files\dotnet\sdk]
  2.1.701 [C:\Program Files\dotnet\sdk]
  2.1.801 [C:\Program Files\dotnet\sdk]
  2.1.802 [C:\Program Files\dotnet\sdk]
  2.1.803 [C:\Program Files\dotnet\sdk]
  2.2.300 [C:\Program Files\dotnet\sdk]
  2.2.301 [C:\Program Files\dotnet\sdk]
  2.2.401 [C:\Program Files\dotnet\sdk]
  2.2.402 [C:\Program Files\dotnet\sdk]
  3.1.201 [C:\Program Files\dotnet\sdk]
  3.1.300-preview-015095 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.13 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.16 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.17 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.8 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.WindowsDesktop.App 3.1.2 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
  Microsoft.WindowsDesktop.App 3.1.3 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

dotnet run --info

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.18363.815 (1909/November2018Update/19H2)
Intel Core i7-8650U CPU 1.90GHz (Kaby Lake R), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.1.300-preview-015095
  [Host] : .NET Core 3.1.3 (CoreCLR 4.700.20.11803, CoreFX 4.700.20.12001), X64 RyuJIT DEBUG

[adamsitnik]

hmm it looks like you are using a preview of 3.1. Is there any chance that you could update to latest stable from https://dotnet.microsoft.com/download/dotnet-core/3.1 and try it one more time?

[JimBobSquarePants]

Yeah, will do. The preview must have gotten installed by my VS Preview.

[JimBobSquarePants]

No difference I'm afraid.

[adamsitnik]

@JimBobSquarePants I was able to reproduce it. It looks that it's a bug in .NET Core 2.2+ : https://github.com/dotnet/runtime/issues/35832

I am going to close this issue here since there is nothing actionable on the BenchmarkDotNet side.

[adamsitnik]

It was not a .NET Core bug, I've sent a fix to https://github.com/AraHaan/GitInformation/pull/39

[JimBobSquarePants]

Ah you’re a legend. Thanks!

[AraHaan]

Ah I see I did not know I messed up on a few small things. What goes to show, even I can cause major performance issues/ bugs that I never experience myself when debugging on my device.

Even I did not know this one either.

[AraHaan]

Also filed this so more people do not get bit like me by the unknowing of things.
https://github.com/dotnet/roslyn-analyzers/issues/3604

[AraHaan]

Alright have since then published as prerelease for now. They will remain at that state for a while until all the work needed is complete.