Runtime: Performance Issues Adding Items To List Is Slower in Core 3 than Core 2.2

Created on 11 Jul 2019 · 8Comments · Source: dotnet/runtime

Below are the benchmark results, using BenchmarkDotnet, to test adding items to a List<> collection. As you can see, while .NET Core 3 is faster than the Clr, it seems the performance is slower when compared to .NET Core 2.2.

Below is the code for the benchmark test:

        [Benchmark(Description = "COLLECTION CREATION: Using List<> & Add()")]
        public List<FakePerson> TestAddingItemsToCollections01()
        {
            var collection = new List<FakePerson>();

            for (var count = 0; count < this.CollectionCount; count++)
            {
                collection.Add(this._fakePersonFixedCollection[count]);
            }

            return collection;
        }

area-System.Collections tenet-performance

Source

RealDotNetDave

All 8 comments

I cannot repro your results. I guess it would be helpful if you share your whole benchmark code.

| Method | Job | Runtime | Toolchain | Mean | Error | StdDev |
|-------------------------------------------- |-------- |-------- |-------------- |---------:|----------:|----------:|
| 'COLLECTION CREATION: Using List<> & Add()' | Default | Core | .NET Core 2.2 | 33.25 us | 0.3665 us | 0.3061 us |
| 'COLLECTION CREATION: Using List<> & Add()' | Clr | Clr | Default | 38.67 us | 0.1460 us | 0.1366 us |
| 'COLLECTION CREATION: Using List<> & Add()' | Core | Core | Default | 33.17 us | 0.9175 us | 1.6309 us |

[ClrJob]
[CoreJob]
[RPlotExporter]
public class ObjectTest
{
    List<FakePerson> _fakePersonFixedCollection = new List<FakePerson>();

    const int IterationCount = 5000;

    [GlobalSetup]
    public void Setup()
    {
        for (int i = 0; i < IterationCount; i++)
        {
            _fakePersonFixedCollection.Add(new FakePerson());
        }
    }

    [Benchmark(Description = "COLLECTION CREATION: Using List<> & Add()")]
    public List<FakePerson> TestAddingItemsToCollections01()
    {
        var collection = new List<FakePerson>();

        for (var count = 0; count < IterationCount; count++)
        {
            collection.Add(_fakePersonFixedCollection[count]);
        }

        return collection;
    }

}
public class FakePerson { }

Symbai on 11 Jul 2019

@adamsitnik could you add your insight here to see if it there is something we can still do for 3.0?

safern on 15 Jul 2019

I am unable to reproduce this regression using both the benchmarks from performance repo and provided here.

git clone https://github.com/dotnet/performance.git
cd performance\src\benchmarks\micro
dotnet run -c Release -f netcoreapp2.2 --filter *AddDefaultSize*.List --join --runtimes netcoreapp2.2 netcoreapp3.0

BenchmarkDotNet=v0.11.3.1003-nightly, OS=Windows 10.0.18362
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET Core SDK=3.0.100-preview7-012697
  [Host]     : .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04), 64bit RyuJIT
  Job-SEHBXU : .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04), 64bit RyuJIT
  Job-ZVECHT : .NET Core 3.0.0-preview7-27826-20 (CoreCLR 4.700.19.32603, CoreFX 4.700.19.32613), 64bit RyuJIT

Runtime=Core  IterationTime=250.0000 ms  MaxIterationCount=20  
MinIterationCount=15  WarmupCount=1

| Type | Method | Toolchain | Count | Mean |
|----------------------- |------- |-------------- |------ |---------:|
| AddDefaultSize<Int32> | List | netcoreapp2.2 | 512 | 1.370 us |
| AddDefaultSize<Int32> | List | netcoreapp3.0 | 512 | 1.370 us |
| | | | | |
| AddDefaultSize<String> | List | netcoreapp2.2 | 512 | 2.746 us |
| AddDefaultSize<String> | List | netcoreapp3.0 | 512 | 2.706 us |

@RealDotNetDave could you please share you OS and hardware info printed by BDN?

adamsitnik on 15 Jul 2019

👍1

Is this what you are looking for:

BenchmarkDotNet=v0.11.5.1116-nightly, OS=Windows 10.0.17763.615 (1809/October2018Update/Redstone5)
Unknown processor
.NET Core SDK=3.0.100-preview6-012264
[Host] : .NET Core 3.0.0-preview6-27804-01 (CoreCLR 4.700.19.30373, CoreFX 4.700.19.30308), 64bit RyuJIT
Job-XDILCI : .NET Core 3.0.0-preview6-27804-01 (CoreCLR 4.700.19.30373, CoreFX 4.700.19.30308), 64bit RyuJIT
Job-ZHCSTT : .NET Framework 4.8 (4.8.3801.0), 64bit RyuJIT

EvaluateOverhead=True Server=True

RealDotNetDave on 15 Jul 2019

Marked as 3.0 as @adamsitnik said offline he was investigating if there was indeed a regression. He can update it further once he has an answer

safern on 18 Jul 2019

The issue here is incomplete; a benchmark is included but it's incomplete, and as a result others who are trying to repro it have to make assumptions about what's being measured and how. There is nothing actionable here at present.

stephentoub on 18 Jul 2019

👍1

Closing. Please reopen if you can provide a complete benchmark.

ericstj on 20 Aug 2019

👍1

Please excuse me, when I was doing the 2.2 vs 3.0 perf comparison I've forgotten to update this issue.

I've run our microbenchmarks for all OSes and hardware architectures and verified that list.Add has not regressed for both value and reference types.

System.Collections.CreateAddAndClear<Int32>.List(Size: 512)

| Conclusion | Base | Diff | Base/Diff | Modality | Operating System | Bit | Processor Name | Base Runtime | Diff Runtime |
| ---------- | --------:| --------:| ---------:| -------- | -------------------- | ----- | ------------------------------------------- | --------------- | --------------------------------- |
| Same | 1252.63 | 1241.69 | 1.01 | | Windows 10.0.18362 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 1461.86 | 1465.58 | 1.00 | | Windows 10.0.18362 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 988.63 | 1020.49 | 0.97 | | Windows 10.0.18362 | 64bit | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Faster | 1947.89 | 1831.79 | 1.06 | | Windows 10.0.18362 | 64bit | AMD Ryzen 7 1800X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 1333.07 | 1279.56 | 1.04 | | Windows 10.0.18362 | 64bit | AMD Ryzen 9 3900X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 1266.76 | 1264.34 | 1.00 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 2145.55 | 2460.53 | 0.87 | several? | ubuntu 16.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 2199.12 | 2330.11 | 0.94 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 1530.82 | 1544.63 | 0.99 | | macOS Mojave 10.14.5 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Faster | 1555.31 | 1415.76 | 1.10 | | Windows 10.0.18362 | 32bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 1778.71 | 1796.94 | 0.99 | | Windows 10.0.18362 | 32bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-17|
| Same | 8183.24 | 8321.25 | 0.98 | | ubuntu 18.04 | 64bit | ARMv8 Processor rev 1 (v8l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 23393.39 | 22385.59 | 1.05 | | ubuntu 14.04 | 32bit | ARMv7 Processor rev 3 (v7l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28374-06|

System.Collections.CreateAddAndClear<String>.List(Size: 512)

| Conclusion | Base | Diff | Base/Diff | Modality | Operating System | Bit | Processor Name | Base Runtime | Diff Runtime |
| ---------- | --------:| --------:| ---------:| -------- | -------------------- | ----- | ------------------------------------------- | --------------- | --------------------------------- |
| Same | 2549.86 | 2493.62 | 1.02 | | Windows 10.0.18362 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 2916.07 | 2931.68 | 0.99 | | Windows 10.0.18362 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 1938.50 | 2208.14 | 0.88 | | Windows 10.0.18362 | 64bit | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Faster | 2860.29 | 2593.74 | 1.10 | | Windows 10.0.18362 | 64bit | AMD Ryzen 7 1800X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 2152.91 | 2117.93 | 1.02 | | Windows 10.0.18362 | 64bit | AMD Ryzen 9 3900X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 2677.66 | 2624.24 | 1.02 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 4754.08 | 5002.66 | 0.95 | | ubuntu 16.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 4906.65 | 4957.92 | 0.99 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 3050.68 | 3071.14 | 0.99 | | macOS Mojave 10.14.5 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 3060.93 | 2925.85 | 1.05 | | Windows 10.0.18362 | 32bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 3440.29 | 3480.56 | 0.99 | | Windows 10.0.18362 | 32bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-17|
| Faster | 32495.30 | 30172.20 | 1.08 | | ubuntu 18.04 | 64bit | ARMv8 Processor rev 1 (v8l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 40730.86 | 52069.16 | 0.78 | several? | ubuntu 14.04 | 32bit | ARMv7 Processor rev 3 (v7l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28374-06|

System.Collections.AddGivenSize<String>.List(Size: 512)

| Conclusion | Base | Diff | Base/Diff | Modality | Operating System | Bit | Processor Name | Base Runtime | Diff Runtime |
| ---------- | --------:| --------:| ---------:| -------- | -------------------- | ----- | ------------------------------------------- | --------------- | --------------------------------- |
| Same | 1918.94 | 1980.05 | 0.97 | | Windows 10.0.18362 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 2129.80 | 2145.93 | 0.99 | | Windows 10.0.18362 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 1315.33 | 1428.20 | 0.92 | | Windows 10.0.18362 | 64bit | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 1843.32 | 1769.67 | 1.04 | | Windows 10.0.18362 | 64bit | AMD Ryzen 7 1800X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 1378.05 | 1364.72 | 1.01 | | Windows 10.0.18362 | 64bit | AMD Ryzen 9 3900X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Faster | 1921.73 | 1779.62 | 1.08 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 2975.40 | 3544.56 | 0.84 | | ubuntu 16.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 3412.12 | 3199.57 | 1.07 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 2009.45 | 2113.28 | 0.95 | | macOS Mojave 10.14.5 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Faster | 2571.46 | 2318.10 | 1.11 | | Windows 10.0.18362 | 32bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Faster | 2929.17 | 2731.92 | 1.07 | | Windows 10.0.18362 | 32bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-17|
| Same | 25008.93 | 23714.23 | 1.05 | | ubuntu 18.04 | 64bit | ARMv8 Processor rev 1 (v8l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 41851.02 | 31163.83 | 1.34 | bimodal | ubuntu 14.04 | 32bit | ARMv7 Processor rev 3 (v7l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28374-06|

System.Collections.AddGivenSize<Int32>.List(Size: 512)

| Conclusion | Base | Diff | Base/Diff | Modality | Operating System | Bit | Processor Name | Base Runtime | Diff Runtime |
| ---------- | --------:| --------:| ---------:| -------- | -------------------- | ----- | ------------------------------------------- | --------------- | --------------------------------- |
| Same | 934.43 | 929.10 | 1.01 | | Windows 10.0.18362 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 1101.42 | 1097.02 | 1.00 | | Windows 10.0.18362 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 706.15 | 716.35 | 0.99 | | Windows 10.0.18362 | 64bit | Intel Core i7-7700 CPU 3.60GHz (Kaby Lake) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 1310.24 | 1287.32 | 1.02 | | Windows 10.0.18362 | 64bit | AMD Ryzen 7 1800X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 974.88 | 952.71 | 1.02 | | Windows 10.0.18362 | 64bit | AMD Ryzen 9 3900X | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-12|
| Same | 927.79 | 955.40 | 0.97 | | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Slower | 1539.80 | 1732.38 | 0.89 | | ubuntu 16.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 1612.43 | 1694.75 | 0.95 | several? | ubuntu 18.04 | 64bit | Intel Xeon CPU E5-2673 v4 2.30GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-01|
| Same | 1120.86 | 1121.49 | 1.00 | | macOS Mojave 10.14.5 | 64bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Faster | 1168.04 | 993.63 | 1.18 | | Windows 10.0.18362 | 32bit | Intel Xeon CPU E5-1650 v4 3.60GHz | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Faster | 1370.08 | 1217.43 | 1.13 | | Windows 10.0.18362 | 32bit | Intel Core i7-5557U CPU 3.10GHz (Broadwell) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28373-17|
| Slower | 5114.72 | 5424.14 | 0.94 | | ubuntu 18.04 | 64bit | ARMv8 Processor rev 1 (v8l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-27919-09|
| Same | 10670.79 | 12921.86 | 0.83 | bimodal | ubuntu 14.04 | 32bit | ARMv7 Processor rev 3 (v7l) | .NET Core 2.2.6 | .NET Core 3.0.0-preview8-28374-06|