Benchmarkdotnet: HardwareCounter.InstructionRetired failing with ArgumentNullException. Build 82

[redknightlois]

I tried to run: https://gist.github.com/redknightlois/7341ae1f13924a1baef49df9415f40af
And got this exception.

// Exception: System.ArgumentNullException: Value cannot be null.
Parameter name: source
   at System.Linq.Enumerable.Select[TSource,TResult](IEnumerable`1 source, Func`2 selector)
   at BenchmarkDotNet.Diagnostics.Windows.PmcStats..ctor(IReadOnlyCollection`1 hardwareCounters)
   at BenchmarkDotNet.Diagnostics.Windows.PmcDiagnoser.GetInitializedStats(Benchmark benchmark)
   at BenchmarkDotNet.Diagnostics.Windows.EtwDiagnoser`1.Start(Process process, Benchmark benchmark)
   at BenchmarkDotNet.Extensions.CommonExtensions.ForEach[T](IList`1 source, Action`1 command)
   at BenchmarkDotNet.Loggers.SynchronousProcessOutputLoggerWithDiagnoser.ProcessInput()
   at BenchmarkDotNet.Toolchains.Executor.Execute(Process process, Benchmark benchmark, SynchronousProcessOutputLoggerWithDiagnoser loggerWithDiagnoser, ILogger logger)
   at BenchmarkDotNet.Toolchains.Executor.Execute(Benchmark benchmark, ILogger logger, String exePath, String workingDirectory, String args, IDiagnoser diagnoser, IResolver resolver)
   at BenchmarkDotNet.Toolchains.Executor.Execute(BuildResult buildResult, Benchmark benchmark, ILogger logger, IResolver resolver, IDiagnoser compositeDiagnoser)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Execute(ILogger logger, Benchmark benchmark, IToolchain toolchain, BuildResult buildResult, IConfig config, IResolver resolver, GcStats& gcStats)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)

Unhandled Exception: System.UnauthorizedAccessException: Access to the path 'G:\Repositories\Performance-bench.git\Course.FullNet\bin\Release\BDN.Generated.exe' is denied.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalDelete(String path, Boolean checkHost)
   at BenchmarkDotNet.Toolchains.Roslyn.Generator.Cleanup(Benchmark benchmark, ArtifactsPaths artifactsPaths)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark benchmark, ILogger logger, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark[] benchmarks, ILogger logger, String title, IConfig config, String rootArtifactsFolderPath, Func`2 toolchainProvider, IResolver resolver)
   at BenchmarkDotNet.Running.BenchmarkRunnerCore.Run(Benchmark[] benchmarks, IConfig config, Func`2 toolchainProvider)
   at BenchmarkDotNet.Running.BenchmarkSwitcher.RunBenchmarks(String[] args, IConfig config)
   at Course.Program.Main(String[] args) in G:\Repositories\Performance-bench.git\Course\Program.cs:line 17

[redknightlois]

I can confirm that still fails on build 10.3.86 ... I am giving a course on performance tomorrow and using BenchmarkDotNet throughout it. It would be awesome to showcase the ability to measure low level CPU behavior. Do you think it could be solved by then? @AndreyAkinshin @adamsitnik @mattwarren

[adamsitnik]

@redknightlois sure, starting investigation right now

[adamsitnik]

@redknightlois I can reproduce it, it's my bug in the configuration. I'll work on a fix now, but in the meantime you can workaround it:

Please comment the per type config: //[HardwareCounters(HardwareCounter.InstructionRetired)]
and extend your job in following way:

Job.Default
    .With(Platform.X64)
    .With(Jit.RyuJit)
    .WithHardwareCounters(HardwareCounter.InstructionRetired)

@AndreyAkinshin I am going to make the counters part of IConfig, not Job.

btw: sample results:

BenchmarkDotNet=v0.10.3.20170407-develop, OS=Windows 10.0.14393
Processor=Intel(R) Core(TM) i7-6600U CPU 2.60GHz, ProcessorCount=4
Frequency=2742191 Hz, Resolution=364.6719 ns, Timer=TSC
  [Host]     : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1637.0
  Job-HNGBTQ : Clr 4.0.30319.42000, 64bit RyuJIT-v4.6.1637.0

HardwareCounters=InstructionRetired  Jit=RyuJit  Platform=X64  

| Method | Mean | StdDev | InstructionRetired/Op | Gen 0 | Allocated |
|------------------- |----------- |---------- |---------------------- |-------- |---------- |
| StackByRef | 2.1916 us | 0.0204 us | 15377 | - | 0 B |
| StackByValue | 15.8732 us | 0.1817 us | 44138 | - | 0 B |
| HeapByConstruction | 5.9767 us | 0.1502 us | 45896 | 22.8014 | 48 kB |
| HeapByReuse | 2.2304 us | 0.0403 us | 15534 | - | 48 B |

[adamsitnik]

@redknightlois fixed, you don't need the workaround anymore. The CI version build number is 0.10.3.87

Good luck with your course tomorrow!

[redknightlois]

Thanks!!! Confirmed working!

[AndreyAkinshin]

@adamsitnik, another awesome bugfix, thanks!

I am going to make the counters part of IConfig, not Job.

It's a very good idea!