Benchmarkdotnet: How does the CsProjNet46Toolchain work?

[ig-sinicyn]

Proof (be sure to run as x64):

    public class Program
    {
        static void Main(string[] args)
        {
            var job = Job.Default.With(new CsProjNet46Toolchain());
            var config = DefaultConfig.Instance.With(job).KeepBenchmarkFiles(true);
            BenchmarkRunner.Run(typeof(Program), config);

        }

        [Benchmark]
        public static void Run() => Thread.SpinWait(1000);

        [Benchmark]
        public static void Run3() => Thread.SpinWait(3000);
    }

output:

Program_Run_Job-XLTLVH.notcs(8,23): error CS0234: The type or namespace name 'Characteristics' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [c:\users\igors\documents\visual studio 2017\Projects\BDN.Proof\BDN.Proof\bin\Release\Program_Run_Job-XLTLVH\BenchmarkDotNet.Autogenerated.csproj]
Program_Run_Job-XLTLVH.notcs(9,23): error CS0234: The type or namespace name 'Environments' does not exist in the namespace 'BenchmarkDotNet' (are you missing an assembly reference?) [c:\users\igors\documents\visual studio 2017\Projects\BDN.Proof\BDN.Proof\bin\Release\Program_Run_Job-XLTLVH\BenchmarkDotNet.Autogenerated.csproj]

BenchmarkDotNet.Autogenerated.csproj:

<Project Sdk="Microsoft.NET.Sdk" ToolsVersion="15.0">
  <PropertyGroup>
    <AssemblyTitle>Program_Run3_Job-XLTLVH</AssemblyTitle>
    <TargetFramework>net46</TargetFramework>
    <AllowUnsafeBlocks>true</AllowUnsafeBlocks>
    <PlatformTarget>x64</PlatformTarget>
    <AssemblyName>Program_Run3_Job-XLTLVH</AssemblyName>
    <OutputType>Exe</OutputType>
    <OutputPath>bin\Release</OutputPath>

  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Program_Run3_Job-XLTLVH.notcs" Exclude="bin\**;obj\**;**\*.xproj;packages\**" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="c:\users\igors\documents\visual studio 2017\Projects\BDN.Proof\BDN.Proof\BDN.Proof.csproj" />
  </ItemGroup>

</Project>

[adamsitnik]

@ig-sinicyn I can't reproduce it, moreover this case is covered by our integration tests which ofc pass.

Are you sure that your BDN.Proof.csproj has reference to BenchmarkDotNet.Core.dll and that it is using any of it's types?

As far as I know BDN does not support static benchmarks except InProcessToolchain, and your methods are static.

[ig-sinicyn]

@adamsitnik

Here's the repro project, BDN.Proof.zip. I've checked it with instance members, no effect.

[adamsitnik]

@ig-sinicyn Ok, now I see what is the problem. You are having old format .csproj which does not work like the new csprojs (dependencies resolved in recursive way).

Old csproj:
A has reference to BDN & B has reference to A => B can't use BDN without adding explicit reference

New csproj:
A has reference to BDN & B has reference to A => B can use BDN without adding explicit reference

This is why it's not a default toolchain for building classic .NET assemblies, this toolchain is designed to be used from .NET Core context, when .NET Core host want to buid a classic .NET assembly.

I am closing this issue, will add a comment to this class summary.

[ig-sinicyn]

@adamsitnik can you please reopen the issue?

It's the same error that was reported multiple times before (#234 and #327, as example) and I hope it will be fixed sooner or later:)

If this is not an option, is there any way to run an outofproc benchmark referencing BenchmarkDotNet.Core only (without reference to the BenchmarkDotNet.Toolchains.Roslyn)?

Roslyn dependency results in +~250 Mb in packages folder which is not an option in some of our scenarios (some of our deployment tests set very hard limits on timing and network bandwith).

[adamsitnik]

@ig-sinicyn I am sorry but I will not reopen this issue. Both RoslynToolchain and CsProjNet46Toolchain have a single responsibility. And they serve their purpose well.

I know that Roslyn dependency is huge, but what we had before (MSBuild) was very error prone. It just did not work for too many scenarios.

What you are asking for is a toolchain, that has no Roslyn dependency and works for the old csprojs. Should it also work for the new csprojs? What about Mono? Should it work for Mono on all OSes? What about .NET 4.6+? That is just a new feature.

I have spent dozens if not hundreds of hours to get the toolchains working for our end users. I have become project.json/xproj/Roslyn/MS Build/dotnet cli expert and I will simply not spend more of my free time to build another toolchain, only because some people don't want to download 250 MB of dependencies for Roslyn.

I don't want to be harsh or mean, I just want you to understand that I have sacrificed a lof of my free time to get it working and I want to keep it working, but I am not willing to spend any more time on creating another toolchain that is not mandatory.

How to get what you want: you should build a toolchain very similar to CsProjNet46Toolchain which adds references to all required assemblies. you can use Assembly.GetReferencedAssemblies() to get the list. I can't do this in CsProjNet46Toolchain because as I said it was designed to work for .NET Core host process, so generating a csproj that targets .NET 4.6 with references to .NET Core assemblies does not make any sense.

Another option would be to add a toolchain based on MSBuild. Something that we had before. It was too error prone for general use, but it might be perfect for your use case.

[ig-sinicyn]

@adamsitnik ok, thanks for explanation!

Will do a custom toolchain, not a major issue.