Runtime: List<>[] 2.4x-2.6x slower on 3.0 than on 2.2 Regression

[yahorsi]

BenchmarkDotNet=v0.11.4, OS=Windows 10.0.17763.379 (1809/October2018Update/Redstone5)
Intel Core i7-8700 CPU 3.20GHz (Coffee Lake), 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.0.100-preview3-010431
[Host] : .NET Core 2.1.9 (CoreCLR 4.6.27414.06, CoreFX 4.6.27415.01), 64bit RyuJIT
Job-FBDUOI : .NET Core 2.2.3 (CoreCLR 4.6.27414.05, CoreFX 4.6.27414.05), 64bit RyuJIT
Job-KSLXBC : .NET Core 3.0.0-preview3-27503-5 (CoreCLR 4.6.27422.72, CoreFX 4.7.19.12807), 64bit RyuJIT

Runtime=Core

| Method | Toolchain | Index | Mean | Error | StdDev | Ratio | RatioSD |
|-------------- |-------------- |------ |----------:|----------:|----------:|------:|--------:|
| AccessByIndex | netcoreapp2.2 | 1 | 0.2279 ns | 0.0196 ns | 0.0183 ns | 0.48 | 0.04 |
| AccessByIndex | netcoreapp3.0 | 1 | 0.4749 ns | 0.0017 ns | 0.0014 ns | 1.00 | 0.00 |
| | | | | | | | |
| AccessByIndex | netcoreapp2.2 | 5 | 0.2198 ns | 0.0294 ns | 0.0275 ns | 0.40 | 0.06 |
| AccessByIndex | netcoreapp3.0 | 5 | 0.5485 ns | 0.0375 ns | 0.0351 ns | 1.00 | 0.00 |
| | | | | | | | |
| AccessByIndex | netcoreapp2.2 | 9 | 0.2040 ns | 0.0025 ns | 0.0021 ns | 0.39 | 0.02 |
| AccessByIndex | netcoreapp3.0 | 9 | 0.5236 ns | 0.0335 ns | 0.0313 ns | 1.00 | 0.00 |

Source Code
https://github.com/yahorsi/ListBenchmarks

[danmosemsft]

Are we missing benchmarks in https://github.com/dotnet/performance/ ?

[mikedn]

I don't see any significant difference between 2.2 and 3.0 on my machine. I'll check the code but the one generated by 3.0 looks pretty much as expected and I doubt that 2.2 did any better.

[mikedn]

With tiered compilation disabled 2.2 and 3.0 generate almost identical code:

; .NET 2.2
00007FFF162D9D40 48 83 EC 28          sub         rsp,28h  
00007FFF162D9D44 48 8B 41 08          mov         rax,qword ptr [rcx+8]  
00007FFF162D9D48 8B 51 10             mov         edx,dword ptr [rcx+10h]  
00007FFF162D9D4B 3B 50 18             cmp         edx,dword ptr [rax+18h]  
00007FFF162D9D4E 73 15                jae         00007FFF162D9D65  
00007FFF162D9D50 48 8B 40 08          mov         rax,qword ptr [rax+8]  
00007FFF162D9D54 3B 50 08             cmp         edx,dword ptr [rax+8]  
00007FFF162D9D57 73 12                jae         00007FFF162D9D6B  
00007FFF162D9D59 48 63 D2             movsxd      rdx,edx  
00007FFF162D9D5C 8B 44 90 10          mov         eax,dword ptr [rax+rdx*4+10h]  
00007FFF162D9D60 48 83 C4 28          add         rsp,28h  
00007FFF162D9D64 C3                   ret  
00007FFF162D9D65 E8 B6 C7 61 5E       call        00007FFF748F6520  
00007FFF162D9D6A CC                   int         3  
00007FFF162D9D6B E8 40 7F 90 5F       call        00007FFF75BE1CB0  
00007FFF162D9D70 CC                   int         3  

; .NET 3.0
00007FFF1F44F650 48 83 EC 28          sub         rsp,28h  
00007FFF1F44F654 90                   nop  
00007FFF1F44F655 48 8B 41 08          mov         rax,qword ptr [rcx+8]  
00007FFF1F44F659 8B 51 10             mov         edx,dword ptr [rcx+10h]  
00007FFF1F44F65C 3B 50 10             cmp         edx,dword ptr [rax+10h]  
00007FFF1F44F65F 73 15                jae         00007FFF1F44F676  
00007FFF1F44F661 48 8B 40 08          mov         rax,qword ptr [rax+8]  
00007FFF1F44F665 3B 50 08             cmp         edx,dword ptr [rax+8]  
00007FFF1F44F668 73 12                jae         00007FFF1F44F67C  
00007FFF1F44F66A 48 63 D2             movsxd      rdx,edx  
00007FFF1F44F66D 8B 44 90 10          mov         eax,dword ptr [rax+rdx*4+10h]  
00007FFF1F44F671 48 83 C4 28          add         rsp,28h  
00007FFF1F44F675 C3                   ret  
00007FFF1F44F676 E8 A5 BD B3 FF       call        00007FFF1EF8B420  
00007FFF1F44F67B CC                   int         3  
00007FFF1F44F67C E8 4F 24 78 5F       call        00007FFF7EBD1AD0  
00007FFF1F44F681 CC                   int         3  

The only difference is a nop at the beginning of the method. Unlike to cause issues.

[yahorsi]

What's the best practices on benchmarks and tiered compilation and is that possible to control it from benchmarks?

[mikedn]

I would expect the benchmarking framework to handle this, it already does a ton of warmup. But it might be useful to re-run the netcoreapp3.0 benchmarks with COMPLUS_TieredCompilation=0, just to be sure.

[yahorsi]

After running bench many times I should say I have no idea what's going on. Results are very volatile. I've created issue https://github.com/dotnet/BenchmarkDotNet/issues/1124
I will be continuing this investigation until I finally understand what's going on.

Quick question. How in the world we could be sure the code produced by tiered jit, or better say re jit is faster? Is that possible to force jit to produce optimised code from the beginnging to avoid rejitting phase?

[mattwarren]

Is that possible to force jit to produce optimised code from the beginnging to avoid rejitting phase?

@yahorsi set COMPlus_TieredCompilation=0 disables tiered compilation, (see https://github.com/dotnet/coreclr/blob/66c05cea421a9f388eaebece2b9e64ce4560c5c5/src/inc/clrconfigvalues.h#L648-L660 for other options)

[yahorsi]

Thank you @mattwar , but do you know how to force tiered jit to fully optimize? E.g. to make the benchmark to compare not optimized code with fully optimized

[danmosemsft]

Is it alignment issues again? @yahorsi are the volatile results somewhat bimodal (two clusters of values rather than random range)? BDN can be configured to tell you this.

[yahorsi]

The outcome for the about day of running benchmark is following:
Currently, there are few issues:

1.
Dotnet Core 3.0 does not allow to explicitly control what tier is used for jitting the code. Or at least if it does that feature is not used by DotnetBenchmark. If we add to that that dotnet benchmark, by default, dynamically computes how many warmum and real invocations/iterations are used to run benchmark we end up in the situation where benchmark result might be very different depending on the tier used to jit the code. And, so, results might be very different between different machines, different using default/explicit iterations count and so on.

2.
Seems default warmum/iterations/invikations used by BenchmarkDotnet should be tuned when running at least on Core 3.0 to make sure every benchmark will be running on the same tire

1. 
   Benchmark dotnet should somehow prevent code being rejitted while executing benchmark

So, it would be really cool to be able to explicitly control how code is jitted for the benchmarks.
That will make benchmarking more stable and will open some cool scenarios like we will be able actually to test the same benchmark but jitted by different tiers and so will explicitly see how perf changes

// cc @adamsitnik @danmosemsft

[tommcdon]

cc @kouvel

[yahorsi]

Additionally, I think there are scenarios when someone might want to explicitly control jit optimisation level. So it make sense to consider adding e.g. switch to the dotnet.exe like usully -o to control it explicitly :)

[AndyAyersMS]

cc @adamsitnik

BDN tries to ensure we are measuring the most performant versions of code. If that is not working in this case, we should understand why.

There are also challenges in reliably measuring very short execution time operations, and that may be the case here -- we should look into exactly what code BDN is timing, and say, whether the jit is spoiling efforts to increase the duration of the timed portion, and whether or not we're timing tier0 code.

@yahorsi if you add in the ETW diagnoser you can look at the jit logs (say, via perfview) and see whether and when the benchmark methods are getting rejitted.

[dzmitry-lahoda]

I see similar issue in my sample https://gitlab.com/dzmitry-lahoda/dotnet-csharp-patterns/blob/master/indexed-memory/Program.cs. I do not have capacity to research what the issue. Hope that will be fixed (may be on BDN level) to allow to run tests.

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.17134.706 (1803/April2018Update/Redstone4)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview3-010431
  [Host] : .NET Core 2.1.6 (CoreCLR 4.6.27019.06, CoreFX 4.6.27019.05), 64bit RyuJIT
  Core   : .NET Core 2.1.6 (CoreCLR 4.6.27019.06, CoreFX 4.6.27019.05), 64bit RyuJIT

Job=Core  Runtime=Core  InvocationCount=1  
UnrollFactor=1  

|                     Method |         N |     Mean |     Error |    StdDev | Ratio | RatioSD |
|--------------------------- |---------- |---------:|----------:|----------:|------:|--------:|
|            RawArrayIndexer | 100000000 | 184.9 ms |  1.988 ms |  1.762 ms |  1.00 |    0.00 |
|           ArrayWrapIndexer | 100000000 | 191.9 ms |  3.633 ms |  3.731 ms |  1.04 |    0.03 |
|          MemoryWrapIndexer | 100000000 | 629.6 ms | 10.946 ms | 10.238 ms |  3.41 |    0.07 |
|        ArrayViaSpanIndexer | 100000000 | 172.3 ms |  3.271 ms |  2.900 ms |  0.93 |    0.01 |
| ArrayWithParamsSpanIndexer | 100000000 | 178.8 ms |  3.551 ms |  3.646 ms |  0.97 |    0.02 |
|   ArrayViaParamSpanIndexer | 100000000 | 182.4 ms |  3.583 ms |  3.680 ms |  0.99 |    0.02 |
|              MemoryIndexer | 100000000 | 630.9 ms |  9.381 ms |  8.316 ms |  3.41 |    0.05 |
|          MemorySpanIndexer | 100000000 | 189.5 ms |  4.622 ms | 13.554 ms |  0.99 |    0.05 |

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.17134.706 (1803/April2018Update/Redstone4)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview3-010431
  [Host] : .NET Core 3.0.0-preview3-27503-5 (CoreCLR 4.6.27422.72, CoreFX 4.7.19.12807), 64bit RyuJIT
  Core   : .NET Core 3.0.0-preview3-27503-5 (CoreCLR 4.6.27422.72, CoreFX 4.7.19.12807), 64bit RyuJIT

Job=Core  Runtime=Core  InvocationCount=1  
UnrollFactor=1  

|                     Method |         N |     Mean |     Error |    StdDev |   Median | Ratio | RatioSD |
|--------------------------- |---------- |---------:|----------:|----------:|---------:|------:|--------:|
|            RawArrayIndexer | 100000000 | 327.2 ms |  6.275 ms |  6.444 ms | 325.5 ms |  1.00 |    0.00 |
|           ArrayWrapIndexer | 100000000 | 516.6 ms |  8.147 ms |  7.222 ms | 515.8 ms |  1.58 |    0.05 |
|          MemoryWrapIndexer | 100000000 | 740.9 ms | 14.807 ms | 14.542 ms | 737.6 ms |  2.26 |    0.05 |
|        ArrayViaSpanIndexer | 100000000 | 461.4 ms |  9.024 ms | 10.031 ms | 463.0 ms |  1.41 |    0.05 |
| ArrayWithParamsSpanIndexer | 100000000 | 182.1 ms | 25.809 ms | 71.517 ms | 142.3 ms |  0.94 |    0.04 |
|   ArrayViaParamSpanIndexer | 100000000 | 159.7 ms |  5.795 ms | 15.164 ms | 156.9 ms |  0.52 |    0.06 |
|              MemoryIndexer | 100000000 | 842.5 ms | 15.796 ms | 14.003 ms | 843.3 ms |  2.57 |    0.05 |
|          MemorySpanIndexer | 100000000 | 144.7 ms |  3.149 ms |  8.184 ms | 142.7 ms |  0.46 |    0.03 |

BenchmarkDotNet=v0.11.5, OS=Windows 10.0.17134.706 (1803/April2018Update/Redstone4)
Intel Core i7-7700HQ CPU 2.80GHz (Kaby Lake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=3.0.100-preview3-010431
  [Host] : .NET Core 2.2.3 (CoreCLR 4.6.27414.05, CoreFX 4.6.27414.05), 64bit RyuJIT
  Core   : .NET Core 2.2.3 (CoreCLR 4.6.27414.05, CoreFX 4.6.27414.05), 64bit RyuJIT

Job=Core  Runtime=Core  InvocationCount=1  
UnrollFactor=1  

|                     Method |         N |     Mean |     Error |    StdDev |   Median | Ratio | RatioSD |
|--------------------------- |---------- |---------:|----------:|----------:|---------:|------:|--------:|
|            RawArrayIndexer | 100000000 | 187.0 ms |  3.715 ms |  4.129 ms | 186.3 ms |  1.00 |    0.00 |
|           ArrayWrapIndexer | 100000000 | 207.9 ms |  4.621 ms | 13.626 ms | 206.1 ms |  1.12 |    0.07 |
|          MemoryWrapIndexer | 100000000 | 703.1 ms | 14.650 ms | 32.156 ms | 695.3 ms |  3.80 |    0.18 |
|        ArrayViaSpanIndexer | 100000000 | 191.4 ms |  5.072 ms | 14.875 ms | 186.6 ms |  1.01 |    0.05 |
| ArrayWithParamsSpanIndexer | 100000000 | 181.1 ms |  3.724 ms |  8.096 ms | 178.5 ms |  0.99 |    0.06 |
|   ArrayViaParamSpanIndexer | 100000000 | 185.2 ms |  3.442 ms |  3.826 ms | 184.3 ms |  0.99 |    0.03 |
|              MemoryIndexer | 100000000 | 640.8 ms | 12.803 ms | 17.948 ms | 635.5 ms |  3.44 |    0.14 |
|          MemorySpanIndexer | 100000000 | 174.9 ms |  2.155 ms |  1.800 ms | 174.8 ms |  0.93 |    0.02 |

[yahorsi]

Closing as this seems to be BDM issue related to how BDM calculates default iterations that does not correctly work with tiered comp.