Benchmarkdotnet: How to run Clr and Core jobs at the same time?

[kmadof]

Hi

First I tried to run Core and Clr jobs on .net core project but in this case only Core job finished successfully.

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17134.472 (1803/April2018Update/Redstone4)
Intel Core i7-5500U CPU 2.40GHz (Broadwell), 1 CPU, 4 logical and 2 physical cores
.NET Core SDK=2.2.100
  [Host] : .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04), 64bit RyuJIT
  Core   : .NET Core 2.2.0 (CoreCLR 4.6.27110.04, CoreFX 4.6.27110.04), 64bit RyuJIT

| Method | Job | Runtime | Mean | Error | StdDev | Median |
|-------- |----- |-------- |---------:|---------:|---------:|---------:|
| Replace | Clr | Clr | NA | NA | NA | NA |
| Replace | Core | Core | 20.38 us | 1.389 us | 4.095 us | 18.77 us |

Benchmarks with issues:
StringBenchmark.Replace: Clr(Runtime=Clr)

When I checked logs and I found message that project is not compatible with net 472:

// Build Exception: Standard output:
Restoring packages for C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparison\bin\Release\netcoreapp2.2\68bc1c9b-f1a8-4730-8f05-3d8e1e81d768\BenchmarkDotNet.Autogenerated.csproj...
Restore completed in 72.18 ms for C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparison\StringPerformanceComparison.csproj.
C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparison\bin\Release\netcoreapp2.2\68bc1c9b-f1a8-4730-8f05-3d8e1e81d768\BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project StringPerformanceComparison is not compatible with net472 (.NETFramework,Version=v4.7.2). Project StringPerformanceComparison supports: netcoreapp2.2 (.NETCoreApp,Version=v2.2)
C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparison\bin\Release\netcoreapp2.2\68bc1c9b-f1a8-4730-8f05-3d8e1e81d768\BenchmarkDotNet.Autogenerated.csproj : error NU1201: Project StringPerformanceComparison is not compatible with net472 (.NETFramework,Version=v4.7.2) / win7-x64. Project StringPerformanceComparison supports: netcoreapp2.2 (.NETCoreApp,Version=v2.2)
Restore failed in 194.61 ms for C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparison\bin\Release\netcoreapp2.2\68bc1c9b-f1a8-4730-8f05-3d8e1e81d768\BenchmarkDotNet.Autogenerated.csproj.

So I tried slightly different approach. I created full .NET framework console app and check what happens. It ended in similar fashion but this time Clr job finished successfully.

BenchmarkDotNet=v0.11.3, OS=Windows 10.0.17134.472 (1803/April2018Update/Redstone4)
Intel Core i7-5500U CPU 2.40GHz (Broadwell), 1 CPU, 4 logical and 2 physical cores
  [Host] : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3260.0
  Clr    : .NET Framework 4.7.2 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.3260.0

| Method | Job | Runtime | Mean | Error | StdDev |
|-------- |----- |-------- |---------:|----------:|---------:|
| Replace | Clr | Clr | 16.49 us | 0.6754 us | 1.981 us |
| Replace | Core | Core | NA | NA | NA |

Benchmarks with issues:
StringBenchmark.Replace: Core(Runtime=Core)

And this time there were confilcts with resolving proper dependencies due to different versons:

// Build Exception: Standard output:
Microsoft (R) Build Engine version 15.9.20+g88f5fadfbe for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

Restore completed in 45.65 ms for C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparisonFull\bin\Release\f30272d9-c3c3-4011-8f75-7264f756e91d\BenchmarkDotNet.Autogenerated.csproj.
C:\Program Files\dotnet\sdk\2.2.100\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [C:\Dev\TheCodeManual\StringPerformance\StringPerformanceComparisonFull\bin\Release\f30272d9-c3c3-4011-8f75-7264f756e91d\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\2.2.100\Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "System.Reflection.Metadata" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed.

Did I do something worng or there is an issue on dotnetbenchmark side?

[adamsitnik]

Hi @kmadof

Project StringPerformanceComparison is not compatible with net472 (.NETFramework,Version=v4.7.2). Project StringPerformanceComparison supports: netcoreapp2.2 (.NETCoreApp,Version=v2.2)

To run projects for .NET and .NET Core the project that defines benchmarks must target both frameworks. In your case it should be:

<TargetFrameworks>netcoreapp2.2;net472</TargetFrameworks>

Please feel free to reopen the issue if it does not help.

[kmadof]

Thanks a lot. It helped!