Benchmarkdotnet: How to run benchmarks against multiple target frameworks?

[kirsan31]

Hi.
.net 4.7.2 console app, BenchmarkDotNet 0.11.5.
I Have following core sdk installed: 2.1.*, 2.2.*, 3.0.100-preview5-011568.
With [CoreJob] I have got this errors:

Log

// Build Error: Standard output:

Standard error:
Time Elapsed 00:00:01.32
1 Error(s)
2 Warning(s)
e6b5072c-baff-45ee-8006-d640f98ea9dc.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview5-011568\Microsoft.Common.CurrentVersion.targets(2114,5): warning MSB3277: Found conflicts between different versions of "System.Threading.Tasks.Extensions" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview5-011568\Microsoft.Common.CurrentVersion.targets(2114,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
Build FAILED.
e6b5072c-baff-45ee-8006-d640f98ea9dc.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview5-011568\Microsoft.Common.CurrentVersion.targets(2114,5): warning MSB3277: Found conflicts between different versions of "System.Threading.Tasks.Extensions" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview5-011568\Microsoft.Common.CurrentVersion.targets(2114,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview5-011568\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(157,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [...Net_performance_test\bin\x64\Release\e6b5072c-baff-45ee-8006-d640f98ea9dc\BenchmarkDotNet.Autogenerated.csproj]
Copyright (C) Microsoft Corporation. All rights reserved.
Microsoft (R) Build Engine version 16.0.462+g62fb89029d for .NET Core

With Job.Default.With(CsProjCoreToolchain.NetCoreApp22) warnings are gone, but it's still trying to use 3.0 preview:

Log

// Decoder_Test.Decode: Job-ZJYYHM(Toolchain=.NET Core 2.2)

Setup power plan (GUID: 8c5e7fda-e8bf-4a96-9a85-a6e23a8c635c FriendlyName: High performance)// Build Error: Standard output:

Standard error:
Time Elapsed 00:00:03.24
1 Error(s)
0 Warning(s)
9419fe1c-fcd6-48c3-a4bc-acf2e56eb564.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\9419fe1c-fcd6-48c3-a4bc-acf2e56eb564\BenchmarkDotNet.Autogenerated.csproj]
Build FAILED.
9419fe1c-fcd6-48c3-a4bc-acf2e56eb564.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\9419fe1c-fcd6-48c3-a4bc-acf2e56eb564\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\3.0.100-preview5-011568\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.RuntimeIdentifierInference.targets(157,5): message NETSDK1057: You are using a preview version of .NET Core. See: https://aka.ms/dotnet-core-preview [...Net_performance_test\bin\x64\Release\9419fe1c-fcd6-48c3-a4bc-acf2e56eb564\BenchmarkDotNet.Autogenerated.csproj]
Copyright (C) Microsoft Corporation. All rights reserved.
Microsoft (R) Build Engine version 16.0.462+g62fb89029d for .NET Core

c# Config = ManualConfig.CreateEmpty().With(Job.Default.With( CsProjCoreToolchain.From( new NetCoreAppSettings( targetFrameworkMoniker: "netcoreapp2.2", runtimeFrameworkVersion: "2.2.3", name: ".NET Core 2.2"))));
doesn't help either.

After uninstalling 3.0 preview - have the same problem :( All errors and warnings are same (except sdk path ):

[CoreJob]

Standard error:
Time Elapsed 00:00:01.36
1 Error(s)
2 Warning(s)
2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\2.2.300\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.Threading.Tasks.Extensions" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\2.2.300\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a\BenchmarkDotNet.Autogenerated.csproj]
Build FAILED.
2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\2.2.300\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.Threading.Tasks.Extensions" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a\BenchmarkDotNet.Autogenerated.csproj]
C:\Program Files\dotnet\sdk\2.2.300\Microsoft.Common.CurrentVersion.targets(2106,5): warning MSB3277: Found conflicts between different versions of "System.Collections.Immutable" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [...Net_performance_test\bin\x64\Release\2a2ab30a-1bd4-44f6-9f30-e6bd825ad85a\BenchmarkDotNet.Autogenerated.csproj]
Copyright (C) Microsoft Corporation. All rights reserved.
Microsoft (R) Build Engine version 16.1.76+g14b0a930a7 for .NET Core

With(CsProjCoreToolchain.NetCoreApp22)

Standard error:
Time Elapsed 00:00:01.43
1 Error(s)
0 Warning(s)
c264ac55-730c-413d-9f38-a0026718766b.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\c264ac55-730c-413d-9f38-a0026718766b\BenchmarkDotNet.Autogenerated.csproj]
Build FAILED.
c264ac55-730c-413d-9f38-a0026718766b.notcs(122,48): error CS0234: The type or namespace name 'Engines' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [...Net_performance_test\bin\x64\Release\c264ac55-730c-413d-9f38-a0026718766b\BenchmarkDotNet.Autogenerated.csproj]
Copyright (C) Microsoft Corporation. All rights reserved.
Microsoft (R) Build Engine version 16.1.76+g14b0a930a7 for .NET Core

[AndreyAkinshin]

@kirsan31 sorry for the late reply. Could you please paste the source of your original csproj file?

[kirsan31]

Here

Net_performance_test.zip

[adamsitnik]

Hi @kirsan31

BenchmarkDotNet can test the performance for multiple target frameworks only if the project that defines benchmarks targets them as well.

So your project which defines benchmarks (Net_performance_test.csproj) has to target .NET 4.7.2, .NET Core 2.1, 2.2 and 3.0 in this case.

Please see https://github.com/dotnet/BenchmarkDotNet/issues/1165#issuecomment-495731010 for more details

[AndreyAkinshin]

@adamsitnik I think we should print a nice message about the situation instead of the call stack; it can be done as a part (or a continuation) of https://github.com/dotnet/BenchmarkDotNet/pull/1188.