Benchmarkdotnet: Split and simplify printed summaries

[ig-sinicyn]

Hi!

I'm finishing the inline test runner and doing some investigations on configs allowing to run tests as fast as it's possible without losing the accuracy.

The current timings for smallest tests are 2..5 sec per benchmark and the profiler shows that 1 sec from that is spent inside the EnvironmentInfo.GetCurrent() method. To be more precise, it's a GetProcessorName() call.
The EnvironmentInfo is not preserved between runs as NUnit test runner creates a new AppDomain for each run.

It will be fine to skip getting the CPU name or caching it somewhere. Process' environment variable will be fine, I guess.

[adamsitnik]

I have some idea how we could approach this: for each benchmark run (call from autogenerated Program.cs) we should display (use) only the data than can be benchmark (job?) specific like Clr and Architecture. Then the exporters could use all details that are the same for all benchmarks: OS, Processor, ProcessorCount etc.

@mattwarren @AndreyAkinshin what do you guys think?

[ig-sinicyn]

@adamsitnik well, all other infos are not taking time at all, so I'd prefer to have them in the output,

[adamsitnik]

@ig-sinicyn Let's say you run single benchmark with LaunchCount = 5. Then you see the output of ToFormattedString 6 times. 5 for each launch and 1 for summary. I mean that we should display the data that is benchmark specific 5 times and give full info in the summary because it does not change in time.

[ig-sinicyn]

@adamsitnik, agreed!

[mattwarren]

Let's say you run single benchmark with LaunchCount = 5. Then you see the output of ToFormattedString 6 times. 5 for each launch and 1 for summary. I mean that we should display the data that is benchmark specific 5 times and give full info in the summary because it does not change in time.

Sounds sensible to me, anything that makes the output readable and as concise as possible is a win.

I assume you are talking about this part, that is repeated for each launch?

// BenchmarkDotNet-Dev=v0.9.7.0+
// OS=Microsoft Windows NT 6.1.7601 Service Pack 1
// Processor=Intel(R) Core(TM) i7-4800MQ CPU 2.70GHz, ProcessorCount=8
// Frequency=2630810 ticks, Resolution=380.1111 ns, Timer=TSC
// CLR=MS.NET 4.0.30319.42000, Arch=64-bit RELEASE [RyuJIT]
// JitModules=clrjit-v4.6.1076.0

[adamsitnik]

@mattwarren yes, exactly!

[adamsitnik]

I have splitted EnvironmentInfo int two: BenchmarkEnvironmentInfo and HostEnvironmentInfo.
Every information that is expensive to obtain is Lazy now (processor name (@ig-sinicyn) and dotnet cli version)
BenchmarkEnvironmentInfo contains data that can be benchmark-specific:

• processor architecture
• configuration
• clr version (Mono, MS.NET, CORE, DNX)
• has attached debugger flag
• is Ryu Jit flag
• is Server GC flag
• is Concurrent GC flag

HostEnvironmentInfo extends BenchmarkEnvironmentInfo with Computer-specific data:

• BenchmarkDotNet version
• operating system
• processor name
• processor count
• chronometer frequency
• hardware timer kind
• Jit modules
• dotnet cli version

Sample result for single launch:

// Launch: 1 / 1
// Benchmark Process Environment Information:
// CLR=CORE, Arch=64-bit RELEASE [RyuJIT]
// GC=Concurrent Workstation
MainWarmup 1: 1 op, 208267.49 ns, 208.2675 us/op

Sample result for summary:

// * Summary *
Host Process Environment Information:
BenchmarkDotNet-Dev=v0.9.7.0+
OS=Windows
Processor=?, ProcessorCount=8
Frequency=2338339 ticks, Resolution=427.6540 ns, Timer=TSC
CLR=CORE, Arch=64-bit RELEASE [RyuJIT]
GC=Concurrent Workstation
JitModules=?
dotnet cli version: 1.0.0-preview1-002702
Type=IntroGarbageCollection  Mode=SingleRun  Toolchain=Core
LaunchCount=1  WarmupCount=1  TargetCount=1